2024-03-26T21:03:34.032 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T21:03:34.032 DEBUG:teuthology.run:Teuthology command: teuthology --description orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} --owner scheduled_gabrioux@teuthology --name gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi --archive /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122 --verbose -- /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/orig.config.yaml 2024-03-26T21:03:34.061 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T21:03:34.146 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122 branch: wip-guits-testing-2024-03-26-0925 description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} email: gabrioux@ibm.com first_in_suite: false job_id: '7624122' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-03-26-0925 ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) mon_bind_msgr2: false sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-2024-03-26-0925 sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 832 sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-03-26-0925 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjfHS/LLm9chp4tTBTNYEzt9Wmx3MTOAbaSjAZ3EqNEQpOeGC9i3M1VGwA/WnEhh0nsYuFbrSoDLvUGYozlMbA= smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9s/bqBbYrYfYK+CQKNC8yzKiot6LieHixZ+p8b9B3iAaOetGRHE5Fp4D39LhtdAYnQfizNBT9ZfLc4L7zbxvM= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_20:31:35 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T21:03:34.147 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa; will attempt to use it 2024-03-26T21:03:34.148 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks 2024-03-26T21:03:34.148 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T21:03:34.150 INFO:teuthology.task.internal:Checking packages... 2024-03-26T21:03:34.175 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '42fc52b86b14d4f88c40d75eb7102e0ab3918f55' 2024-03-26T21:03:34.175 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:03:34.175 INFO:teuthology.packaging:ref: None 2024-03-26T21:03:34.175 INFO:teuthology.packaging:tag: None 2024-03-26T21:03:34.175 INFO:teuthology.packaging:branch: wip-guits-testing-2024-03-26-0925 2024-03-26T21:03:34.176 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:03:34.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-03-26-0925 2024-03-26T21:03:34.392 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2484.g42fc52b8 2024-03-26T21:03:34.393 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T21:03:34.400 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T21:03:34.400 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T21:03:34.420 INFO:teuthology.task.internal:Saving configuration 2024-03-26T21:03:34.433 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T21:03:34.439 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T21:03:34.459 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 20:58:07.908477', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjfHS/LLm9chp4tTBTNYEzt9Wmx3MTOAbaSjAZ3EqNEQpOeGC9i3M1VGwA/WnEhh0nsYuFbrSoDLvUGYozlMbA='} 2024-03-26T21:03:34.481 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 20:58:07.906794', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9s/bqBbYrYfYK+CQKNC8yzKiot6LieHixZ+p8b9B3iAaOetGRHE5Fp4D39LhtdAYnQfizNBT9ZfLc4L7zbxvM='} 2024-03-26T21:03:34.481 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T21:03:34.487 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T21:03:34.487 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T21:03:34.487 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T21:03:34.555 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb4818290d0>, signals=[15]) 2024-03-26T21:03:34.555 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T21:03:34.562 INFO:teuthology.task.internal:Opening connections... 2024-03-26T21:03:34.563 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-03-26T21:03:34.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:03:34.674 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-03-26T21:03:34.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:03:34.765 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T21:03:34.772 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-03-26T21:03:34.790 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-03-26T21:03:34.790 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-03-26T21:03:34.847 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-03-26T21:03:34.847 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2024-03-26T21:03:34.847 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-03-26T21:03:34.847 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-03-26T21:03:34.847 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2024-03-26T21:03:34.847 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2024-03-26T21:03:34.847 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T21:03:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-03-26T21:03:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2024-03-26T21:03:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T21:03:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-03-26T21:03:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T21:03:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T21:03:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T21:03:34.849 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-03-26T21:03:34.874 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-03-26T21:03:34.893 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-03-26T21:03:34.893 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-03-26T21:03:34.949 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2024-03-26T21:03:34.949 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="9" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="9" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el9" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:LOGO="fedora-logo-icon" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T21:03:34.950 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T21:03:34.951 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-03-26T21:03:34.973 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T21:03:34.983 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T21:03:34.989 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T21:03:34.989 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-03-26T21:03:34.992 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-03-26T21:03:35.008 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T21:03:35.014 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T21:03:35.015 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-03-26T21:03:35.049 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-03-26T21:03:35.155 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T21:03:35.210 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T21:03:35.227 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T21:03:35.227 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T21:03:35.228 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T21:03:35.228 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T21:03:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:03:35.247 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-03-26T21:03:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:03:35.248 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-03-26T21:03:35.305 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-361.el9.x86_64 2024-03-26T21:03:35.305 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.14.0-361.el9.x86_64 2024-03-26T21:03:35.306 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-03-26T21:03:35.307 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-361.el9.x86_64 2024-03-26T21:03:35.307 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.14.0-361.el9.x86_64 2024-03-26T21:03:35.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum install -y kernel 2024-03-26T21:03:37.509 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-26T21:03:37.584 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T21:03:39.192 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-26T21:03:39.209 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-26T21:03:40.108 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-26T21:03:42.693 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-26T21:03:47.421 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.1 MB 00:01 2024-03-26T21:03:48.614 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 873 kB/s | 8.0 MB 00:09 2024-03-26T21:03:48.986 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-26T21:03:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-03-26T21:03:50.255 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-26T21:03:54.465 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.1 MB 00:00 2024-03-26T21:03:55.305 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T21:03:55.916 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-03-26T21:03:57.034 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-03-26T21:03:57.478 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T21:03:57.478 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T21:03:57.624 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:03:57.634 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:03:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-26T21:03:57.634 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:03:57.634 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout:Install 4 Packages 2024-03-26T21:03:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:03:57.636 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 76 M 2024-03-26T21:03:57.636 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 123 M 2024-03-26T21:03:57.636 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:03:58.014 INFO:teuthology.orchestra.run.smithi138.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 194 kB/s | 66 kB 00:00 2024-03-26T21:04:01.900 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-26T21:04:02.077 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:01 ago on Tue 26 Mar 2024 09:04:01 PM UTC. 2024-03-26T21:04:04.065 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T21:04:04.065 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T21:04:04.148 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:04:04.158 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:04:04.158 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-03-26T21:04:04.158 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:04:04.158 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:04:04.158 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T21:04:04.158 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:04:04.158 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 76 M 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 123 M 2024-03-26T21:04:04.159 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:04:04.368 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 391 kB/s | 66 kB 00:00 2024-03-26T21:04:05.934 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 8.6 MB/s | 15 MB 00:01 2024-03-26T21:04:06.993 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 10 MB/s | 27 MB 00:02 2024-03-26T21:04:07.285 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 11 MB/s | 33 MB 00:03 2024-03-26T21:04:07.285 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:04:07.285 INFO:teuthology.orchestra.run.smithi088.stdout:Total 24 MB/s | 76 MB 00:03 2024-03-26T21:04:07.950 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:04:08.304 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:04:08.305 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:04:09.313 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:04:09.313 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:04:10.238 INFO:teuthology.orchestra.run.smithi138.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.2 MB/s | 15 MB 00:12 2024-03-26T21:04:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:04:12.460 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:04:12.534 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:04:15.261 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:04:15.387 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:04:20.064 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:04:20.103 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:28.286 INFO:teuthology.orchestra.run.smithi138.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.1 MB/s | 33 MB 00:30 2024-03-26T21:04:31.319 INFO:teuthology.orchestra.run.smithi138.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 845 kB/s | 27 MB 00:33 2024-03-26T21:04:31.319 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:04:31.319 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.3 MB/s | 76 MB 00:33 2024-03-26T21:04:31.988 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:04:32.354 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:04:32.354 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:04:33.363 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:04:33.364 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:04:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:04:36.479 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:04:36.564 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:04:39.304 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:04:39.442 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:04:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:04:44.259 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:49.023 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:54.255 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:54.255 INFO:teuthology.orchestra.run.smithi088.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T21:04:54.255 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:04:54.299 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:54.729 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:54.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:04:54.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:04:54.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:04:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:04:55.162 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:04:55.162 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:04:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T21:04:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T21:04:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T21:04:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T21:04:55.163 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:04:55.163 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:04:55.488 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:04:55.599 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:04:55.599 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:04:55.599 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T21:04:55.599 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T21:04:55.599 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T21:04:55.599 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T21:04:55.599 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T21:04:55.600 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:04:55.679 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:04:55.679 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:04:55.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.06 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 68.el9 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Group : Unspecified 2024-03-26T21:04:55.713 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 8264254 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : builder@centos.org 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T21:04:55.714 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T21:04:55.715 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T21:04:55.715 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-03-26T21:04:55.715 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:04:55.715 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T21:04:55.716 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T21:04:55.716 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 5.14.0-432.el9.x86_64 2024-03-26T21:04:55.716 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T21:04:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-03-26T21:04:57.728 INFO:teuthology.orchestra.run.smithi088.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T21:04:57.756 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-03-26T21:04:57.759 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-03-26T21:04:57.774 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.TZUpPFdq2j 2024-03-26T21:04:57.774 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TZUpPFdq2j 2024-03-26T21:04:57.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.TZUpPFdq2j 2024-03-26T21:04:58.013 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.TZUpPFdq2j is 6KB 2024-03-26T21:04:58.061 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.TZUpPFdq2j 2024-03-26T21:04:58.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T21:04:58.142 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T21:04:58.142 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T21:04:58.142 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T21:04:58.142 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T21:04:58.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T21:04:58.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-03-26T21:05:18.570 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:05:18.570 INFO:teuthology.orchestra.run.smithi138.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T21:05:18.570 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:05:18.614 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:05:18.974 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:05:18.974 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:05:18.974 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:05:18.974 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:05:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:05:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:05:19.410 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:05:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T21:05:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T21:05:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T21:05:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T21:05:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:05:19.410 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:05:19.734 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:05:19.837 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:05:19.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:05:19.837 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T21:05:19.837 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T21:05:19.838 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T21:05:19.838 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T21:05:19.838 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T21:05:19.838 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:05:19.919 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:05:19.919 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:05:19.919 DEBUG:teuthology.orchestra.run.smithi138:> sudo rpm -qi grub2-tools 2024-03-26T21:05:19.957 INFO:teuthology.orchestra.run.smithi138.stdout:Name : grub2-tools 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Epoch : 1 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Version : 2.06 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Release : 68.el9 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: x86_64 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Group : Unspecified 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Size : 8264254 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:License : GPLv3+ 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T21:05:19.958 INFO:teuthology.orchestra.run.smithi138.stdout:Packager : builder@centos.org 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:Vendor : CentOS 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:Summary : Support tools for GRUB. 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:Description : 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:hardware devices. 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:05:19.959 INFO:teuthology.orchestra.run.smithi138.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T21:05:19.960 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T21:05:19.960 INFO:teuthology.task.kernel:Updating grub on smithi138 to boot 5.14.0-432.el9.x86_64 2024-03-26T21:05:19.960 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T21:05:20.553 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2024-03-26T21:05:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T21:05:22.002 INFO:teuthology.orchestra.run.smithi138.stderr:done 2024-03-26T21:05:22.006 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-03-26T21:05:22.021 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.1WvXsaGCNY 2024-03-26T21:05:22.022 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1WvXsaGCNY 2024-03-26T21:05:22.090 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.1WvXsaGCNY 2024-03-26T21:05:22.231 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.1WvXsaGCNY is 6KB 2024-03-26T21:05:22.280 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.1WvXsaGCNY 2024-03-26T21:05:22.294 DEBUG:teuthology.orchestra.run.smithi138:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T21:05:22.375 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T21:05:22.375 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T21:05:22.375 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T21:05:22.375 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T21:05:22.376 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T21:05:22.524 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2024-03-26T21:05:28.295 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T21:05:28.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-26T21:05:28.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:05:46.783 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-03-26T21:05:52.530 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T21:05:52.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-26T21:05:52.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:05:55.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-26T21:05:55.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:05:58.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-03-26T21:06:10.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-26T21:06:10.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-26T21:06:10.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:06:13.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-03-26T21:06:19.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-26T21:06:19.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:06:28.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-26T21:06:28.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:07:00.852 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-03-26T21:07:01.271 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-26T21:07:01.271 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T21:07:01.271 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-03-26T21:07:01.287 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-432.el9.x86_64 2024-03-26T21:07:01.288 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T21:07:01.288 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T21:07:01.288 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T21:07:02.288 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T21:07:02.289 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T21:07:02.347 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-03-26T21:07:02.389 DEBUG:teuthology.parallel:result is None 2024-03-26T21:07:19.856 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T21:07:31.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-26T21:07:31.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:07:32.223 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-03-26T21:07:32.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-26T21:07:32.625 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T21:07:32.625 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-03-26T21:07:32.641 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-432.el9.x86_64 2024-03-26T21:07:32.641 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T21:07:32.641 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T21:07:32.641 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T21:07:33.642 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T21:07:33.643 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T21:07:33.714 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2024-03-26T21:07:33.751 DEBUG:teuthology.parallel:result is None 2024-03-26T21:07:33.752 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T21:07:33.760 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T21:07:33.760 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T21:07:33.763 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T21:07:33.781 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T21:07:33.792 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T21:07:33.799 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T21:07:33.799 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T21:07:33.822 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T21:07:34.086 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T21:07:34.093 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T21:07:34.093 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T21:07:34.096 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T21:07:34.131 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:07:34.134 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:07:34.146 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:07:34.148 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:07:34.150 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T21:07:34.157 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T21:07:34.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T21:07:34.190 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T21:07:34.220 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T21:07:34.230 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T21:07:34.230 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T21:07:34.262 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T21:07:34.278 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:07:34.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:07:34.455 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:07:34.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:07:34.540 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:07:34.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T21:07:34.608 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:07:34.650 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:07:34.702 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:07:34.784 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:07:34.829 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:07:34.829 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T21:07:34.899 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-03-26T21:07:34.902 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-03-26T21:07:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T21:07:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T21:07:35.265 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T21:07:35.273 INFO:teuthology.task.internal:Starting timer... 2024-03-26T21:07:35.273 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T21:07:35.343 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T21:07:35.353 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T21:07:35.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-03-26T21:07:35.419 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-03-26T21:07:35.421 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2024-03-26T21:07:35.474 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2024-03-26T21:07:35.476 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T21:07:35.476 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-03-26T21:07:35.509 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-03-26T21:07:35.509 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2024-03-26T21:07:35.537 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2024-03-26T21:07:35.537 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi138.front.sepia.ceph.com': 'permissive'} 2024-03-26T21:07:35.537 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T21:07:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:07:35.580 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T21:07:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:07:35.609 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T21:07:35.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-03-26T21:07:35.651 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2024-03-26T21:07:35.688 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T21:07:35.699 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T21:07:35.717 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T21:07:35.718 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T21:12:04.057 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2024-03-26T21:12:04.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-26T21:12:04.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:12:04.151 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-03-26T21:12:04.234 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-26T21:12:04.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-26T21:12:04.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:12:04.322 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-03-26T21:12:04.396 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-26T21:12:04.397 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T21:12:04.407 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T21:12:04.407 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T21:12:04.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T21:12:04.410 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T21:12:04.410 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T21:12:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T21:12:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T21:12:04.466 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T21:12:04.483 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T21:12:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-03-26T21:12:04.500 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-03-26T21:12:04.512 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2024-03-26T21:12:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T21:12:04.526 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2024-03-26T21:12:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T21:12:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T21:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T21:12:04.585 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-03-26T21:12:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T21:12:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-03-26T21:12:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.588 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.589 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.612 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-03-26T21:12:04.616 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T21:12:04.616 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-03-26T21:12:04.616 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.616 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.616 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.616 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:12:04.617 INFO:teuthology.run_tasks:Running task pexec... 2024-03-26T21:12:04.627 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-26T21:12:04.627 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi088.front.sepia.ceph.com 2024-03-26T21:12:04.627 DEBUG:teuthology.orchestra.run.smithi088:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T21:12:04.628 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi138.front.sepia.ceph.com 2024-03-26T21:12:04.628 DEBUG:teuthology.orchestra.run.smithi138:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T21:12:05.208 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:47 ago on Tue 26 Mar 2024 09:11:18 PM UTC. 2024-03-26T21:12:05.234 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 09:11:01 PM UTC. 2024-03-26T21:12:05.354 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:12:05.367 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:12:05.368 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:12:05.368 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:12:05.368 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.1 M 2024-03-26T21:12:05.368 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 9.8 M 2024-03-26T21:12:05.368 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:12:05.379 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:12:05.380 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:12:05.381 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:12:05.382 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.1 M 2024-03-26T21:12:05.382 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 9.8 M 2024-03-26T21:12:05.382 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:12:05.891 INFO:teuthology.orchestra.run.smithi138.stdout:runc-1.1.12-2.el9.x86_64.rpm 13 MB/s | 3.1 MB 00:00 2024-03-26T21:12:05.892 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:12:05.892 INFO:teuthology.orchestra.run.smithi138.stdout:Total 6.1 MB/s | 3.1 MB 00:00 2024-03-26T21:12:05.927 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:12:05.933 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:12:05.933 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:12:06.012 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:12:06.012 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:12:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:12:06.319 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T21:12:06.323 INFO:teuthology.orchestra.run.smithi088.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.6 MB/s | 3.1 MB 00:00 2024-03-26T21:12:06.324 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:12:06.324 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.3 MB/s | 3.1 MB 00:00 2024-03-26T21:12:06.360 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:12:06.366 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:12:06.366 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:12:06.447 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:12:06.448 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:12:06.652 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:12:06.751 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T21:12:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T21:12:07.099 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T21:12:07.099 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:12:07.099 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:12:07.099 INFO:teuthology.orchestra.run.smithi138.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T21:12:07.099 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:12:07.099 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:12:07.286 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T21:12:07.355 DEBUG:teuthology.parallel:result is None 2024-03-26T21:12:07.583 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T21:12:07.584 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:12:07.584 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:12:07.584 INFO:teuthology.orchestra.run.smithi088.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T21:12:07.584 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:12:07.584 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:12:07.846 DEBUG:teuthology.parallel:result is None 2024-03-26T21:12:07.846 INFO:teuthology.run_tasks:Running task install... 2024-03-26T21:12:07.854 DEBUG:teuthology.task.install:project ceph 2024-03-26T21:12:07.855 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'}, 'extra_packages': ['cephadm']} 2024-03-26T21:12:07.855 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'} 2024-03-26T21:12:07.855 INFO:teuthology.task.install:Using flavor: default 2024-03-26T21:12:07.861 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T21:12:07.861 INFO:teuthology.task.install:extra packages: [] 2024-03-26T21:12:07.861 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'tag': None, 'wait_for_package': False} 2024-03-26T21:12:07.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:12:07.864 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'tag': None, 'wait_for_package': False} 2024-03-26T21:12:07.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:12:08.007 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/ 2024-03-26T21:12:08.008 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2484.g42fc52b8 2024-03-26T21:12:08.078 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/ 2024-03-26T21:12:08.078 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2484.g42fc52b8 2024-03-26T21:12:08.177 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T21:12:08.177 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:12:08.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T21:12:08.186 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T21:12:08.186 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:12:08.186 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T21:12:08.204 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T21:12:08.204 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T21:12:08.213 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T21:12:08.213 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T21:12:08.272 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T21:12:08.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T21:12:08.352 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T21:12:08.360 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T21:12:08.417 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2024-03-26T21:12:08.419 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2024-03-26T21:12:08.440 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-03-26T21:12:08.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-03-26T21:12:08.660 INFO:teuthology.orchestra.run.smithi138.stdout:74 files removed 2024-03-26T21:12:08.683 INFO:teuthology.orchestra.run.smithi088.stdout:74 files removed 2024-03-26T21:12:08.687 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw 2024-03-26T21:12:08.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw 2024-03-26T21:12:09.405 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 199 kB/s | 83 kB 00:00 2024-03-26T21:12:09.463 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 182 kB/s | 83 kB 00:00 2024-03-26T21:12:09.731 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 42 kB/s | 12 kB 00:00 2024-03-26T21:12:09.847 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-03-26T21:12:10.082 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-26T21:12:10.197 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-26T21:12:10.633 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-26T21:12:10.709 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T21:12:12.050 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-26T21:12:12.151 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-26T21:12:13.169 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-26T21:12:13.641 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-26T21:12:27.433 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-26T21:12:31.483 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-26T21:12:33.226 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-26T21:12:35.880 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 913 kB/s | 19 MB 00:21 2024-03-26T21:12:36.094 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-26T21:12:41.069 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-26T21:12:43.572 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 9.8 M 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.4 M 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T21:12:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.4 M 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 21 M 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 25 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 746 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 500 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.1 M 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T21:12:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 45 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 134 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 162 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 321 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 298 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 99 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout:Installing weak dependencies: 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:12:43.577 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:12:43.578 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:12:43.578 INFO:teuthology.orchestra.run.smithi138.stdout:Install 30 Packages 2024-03-26T21:12:43.578 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 2 Packages 2024-03-26T21:12:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:12:43.578 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 59 M 2024-03-26T21:12:43.578 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:12:44.645 INFO:teuthology.orchestra.run.smithi138.stdout:(1/32): ceph-base-19.0.0-2484.g42fc52b8.el9.x86 6.8 MB/s | 5.4 MB 00:00 2024-03-26T21:12:44.745 INFO:teuthology.orchestra.run.smithi138.stdout:(2/32): ceph-selinux-19.0.0-2484.g42fc52b8.el9. 253 kB/s | 25 kB 00:00 2024-03-26T21:12:44.854 INFO:teuthology.orchestra.run.smithi138.stdout:(3/32): libcephfs2-19.0.0-2484.g42fc52b8.el9.x8 6.7 MB/s | 746 kB 00:00 2024-03-26T21:12:44.954 INFO:teuthology.orchestra.run.smithi138.stdout:(4/32): libradosstriper1-19.0.0-2484.g42fc52b8. 4.9 MB/s | 500 kB 00:00 2024-03-26T21:12:45.346 INFO:teuthology.orchestra.run.smithi138.stdout:(5/32): ceph-common-19.0.0-2484.g42fc52b8.el9.x 14 MB/s | 21 MB 00:01 2024-03-26T21:12:45.396 INFO:teuthology.orchestra.run.smithi138.stdout:(6/32): python3-ceph-argparse-19.0.0-2484.g42fc 905 kB/s | 45 kB 00:00 2024-03-26T21:12:45.471 INFO:teuthology.orchestra.run.smithi138.stdout:(7/32): librgw2-19.0.0-2484.g42fc52b8.el9.x86_6 9.9 MB/s | 5.1 MB 00:00 2024-03-26T21:12:45.497 INFO:teuthology.orchestra.run.smithi138.stdout:(8/32): python3-ceph-common-19.0.0-2484.g42fc52 1.3 MB/s | 134 kB 00:00 2024-03-26T21:12:45.522 INFO:teuthology.orchestra.run.smithi138.stdout:(9/32): python3-cephfs-19.0.0-2484.g42fc52b8.el 3.1 MB/s | 162 kB 00:00 2024-03-26T21:12:45.548 INFO:teuthology.orchestra.run.smithi138.stdout:(10/32): python3-rados-19.0.0-2484.g42fc52b8.el 6.3 MB/s | 321 kB 00:00 2024-03-26T21:12:45.573 INFO:teuthology.orchestra.run.smithi138.stdout:(11/32): python3-rbd-19.0.0-2484.g42fc52b8.el9. 5.8 MB/s | 298 kB 00:00 2024-03-26T21:12:45.598 INFO:teuthology.orchestra.run.smithi138.stdout:(12/32): python3-rgw-19.0.0-2484.g42fc52b8.el9. 1.9 MB/s | 99 kB 00:00 2024-03-26T21:12:45.748 INFO:teuthology.orchestra.run.smithi138.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-26T21:12:45.774 INFO:teuthology.orchestra.run.smithi138.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 528 kB/s | 106 kB 00:00 2024-03-26T21:12:45.799 INFO:teuthology.orchestra.run.smithi138.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-26T21:12:45.866 INFO:teuthology.orchestra.run.smithi138.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-26T21:12:45.891 INFO:teuthology.orchestra.run.smithi138.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-26T21:12:45.916 INFO:teuthology.orchestra.run.smithi138.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-03-26T21:12:46.059 INFO:teuthology.orchestra.run.smithi138.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 253 kB/s | 36 kB 00:00 2024-03-26T21:12:46.184 INFO:teuthology.orchestra.run.smithi138.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 755 kB/s | 221 kB 00:00 2024-03-26T21:12:46.209 INFO:teuthology.orchestra.run.smithi138.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 315 kB/s | 47 kB 00:00 2024-03-26T21:12:46.668 INFO:teuthology.orchestra.run.smithi138.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 639 kB/s | 309 kB 00:00 2024-03-26T21:12:46.768 INFO:teuthology.orchestra.run.smithi138.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-03-26T21:12:46.868 INFO:teuthology.orchestra.run.smithi138.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-26T21:12:46.977 INFO:teuthology.orchestra.run.smithi138.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2024-03-26T21:12:47.061 INFO:teuthology.orchestra.run.smithi138.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-26T21:12:47.169 INFO:teuthology.orchestra.run.smithi138.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-26T21:12:47.194 INFO:teuthology.orchestra.run.smithi138.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-26T21:12:47.213 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 796 kB/s | 6.1 MB 00:07 2024-03-26T21:12:47.370 INFO:teuthology.orchestra.run.smithi138.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-26T21:12:47.512 INFO:teuthology.orchestra.run.smithi138.stdout:(30/32): librados2-19.0.0-2484.g42fc52b8.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-26T21:12:47.687 INFO:teuthology.orchestra.run.smithi138.stdout:(31/32): librbd1-19.0.0-2484.g42fc52b8.el9.x86_ 9.9 MB/s | 3.1 MB 00:00 2024-03-26T21:12:48.170 INFO:teuthology.orchestra.run.smithi138.stdout:(32/32): ceph-radosgw-19.0.0-2484.g42fc52b8.el9 2.3 MB/s | 9.8 MB 00:04 2024-03-26T21:12:48.174 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:12:48.175 INFO:teuthology.orchestra.run.smithi138.stdout:Total 13 MB/s | 59 MB 00:04 2024-03-26T21:12:48.208 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:12:48.246 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:12:48.246 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:12:48.721 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:12:48.722 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:12:49.056 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-26T21:12:50.154 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:12:50.237 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T21:12:50.267 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T21:12:50.292 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T21:12:50.323 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:12:50.341 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:12:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:12:50.540 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T21:12:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:12:50.963 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T21:12:50.994 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:12:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:12:51.158 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 8/34 2024-03-26T21:12:51.187 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:12:51.249 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:12:51.278 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T21:12:51.321 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 11/34 2024-03-26T21:12:51.390 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:12:51.414 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:12:51.478 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:12:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T21:12:51.638 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T21:12:51.673 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T21:12:51.712 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T21:12:51.737 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T21:12:51.766 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T21:12:51.798 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T21:12:52.117 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T21:12:52.154 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:12:52.235 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:12:52.274 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 23/34 2024-03-26T21:12:52.320 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T21:12:52.779 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T21:12:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:12:52.885 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:12:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 27/34 2024-03-26T21:12:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 28/34 2024-03-26T21:12:55.163 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:12:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:12:55.447 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-26T21:12:55.764 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:12:55.800 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T21:12:55.900 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T21:12:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:12:55.937 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:12:57.943 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 9.8 M 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.4 M 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options 2024-03-26T21:12:57.945 INFO:teuthology.orchestra.run.smithi088.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.4 M 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 21 M 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 25 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 746 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 500 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.1 M 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T21:12:57.946 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 45 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 134 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 162 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 321 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 298 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 99 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-03-26T21:12:57.947 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout:Install 30 Packages 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 59 M 2024-03-26T21:12:57.948 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:12:59.049 INFO:teuthology.orchestra.run.smithi088.stdout:(1/32): ceph-radosgw-19.0.0-2484.g42fc52b8.el9. 11 MB/s | 9.8 MB 00:00 2024-03-26T21:12:59.090 INFO:teuthology.orchestra.run.smithi088.stdout:(2/32): ceph-selinux-19.0.0-2484.g42fc52b8.el9. 608 kB/s | 25 kB 00:00 2024-03-26T21:12:59.232 INFO:teuthology.orchestra.run.smithi088.stdout:(3/32): libcephfs2-19.0.0-2484.g42fc52b8.el9.x8 5.1 MB/s | 746 kB 00:00 2024-03-26T21:12:59.316 INFO:teuthology.orchestra.run.smithi088.stdout:(4/32): libradosstriper1-19.0.0-2484.g42fc52b8. 5.9 MB/s | 500 kB 00:00 2024-03-26T21:12:59.808 INFO:teuthology.orchestra.run.smithi088.stdout:(5/32): librgw2-19.0.0-2484.g42fc52b8.el9.x86_6 10 MB/s | 5.1 MB 00:00 2024-03-26T21:12:59.858 INFO:teuthology.orchestra.run.smithi088.stdout:(6/32): python3-ceph-argparse-19.0.0-2484.g42fc 904 kB/s | 45 kB 00:00 2024-03-26T21:12:59.917 INFO:teuthology.orchestra.run.smithi088.stdout:(7/32): python3-ceph-common-19.0.0-2484.g42fc52 2.2 MB/s | 134 kB 00:00 2024-03-26T21:12:59.992 INFO:teuthology.orchestra.run.smithi088.stdout:(8/32): ceph-base-19.0.0-2484.g42fc52b8.el9.x86 2.9 MB/s | 5.4 MB 00:01 2024-03-26T21:13:00.017 INFO:teuthology.orchestra.run.smithi088.stdout:(9/32): python3-cephfs-19.0.0-2484.g42fc52b8.el 1.6 MB/s | 162 kB 00:00 2024-03-26T21:13:00.059 INFO:teuthology.orchestra.run.smithi088.stdout:(10/32): python3-rados-19.0.0-2484.g42fc52b8.el 4.7 MB/s | 321 kB 00:00 2024-03-26T21:13:00.084 INFO:teuthology.orchestra.run.smithi088.stdout:(11/32): python3-rbd-19.0.0-2484.g42fc52b8.el9. 4.3 MB/s | 298 kB 00:00 2024-03-26T21:13:00.151 INFO:teuthology.orchestra.run.smithi088.stdout:(12/32): python3-rgw-19.0.0-2484.g42fc52b8.el9. 1.1 MB/s | 99 kB 00:00 2024-03-26T21:13:00.235 INFO:teuthology.orchestra.run.smithi088.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-26T21:13:00.260 INFO:teuthology.orchestra.run.smithi088.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2024-03-26T21:13:00.285 INFO:teuthology.orchestra.run.smithi088.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-26T21:13:00.336 INFO:teuthology.orchestra.run.smithi088.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-26T21:13:00.361 INFO:teuthology.orchestra.run.smithi088.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-26T21:13:00.386 INFO:teuthology.orchestra.run.smithi088.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 823 kB/s | 41 kB 00:00 2024-03-26T21:13:00.436 INFO:teuthology.orchestra.run.smithi088.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 720 kB/s | 36 kB 00:00 2024-03-26T21:13:00.462 INFO:teuthology.orchestra.run.smithi088.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-03-26T21:13:00.679 INFO:teuthology.orchestra.run.smithi088.stdout:(21/32): ceph-common-19.0.0-2484.g42fc52b8.el9. 8.0 MB/s | 21 MB 00:02 2024-03-26T21:13:00.729 INFO:teuthology.orchestra.run.smithi088.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 162 kB/s | 47 kB 00:00 2024-03-26T21:13:00.954 INFO:teuthology.orchestra.run.smithi088.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-03-26T21:13:01.013 INFO:teuthology.orchestra.run.smithi088.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 561 kB/s | 309 kB 00:00 2024-03-26T21:13:01.038 INFO:teuthology.orchestra.run.smithi088.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-03-26T21:13:01.089 INFO:teuthology.orchestra.run.smithi088.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-26T21:13:01.197 INFO:teuthology.orchestra.run.smithi088.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-26T21:13:01.222 INFO:teuthology.orchestra.run.smithi088.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-26T21:13:01.323 INFO:teuthology.orchestra.run.smithi088.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-03-26T21:13:01.356 INFO:teuthology.orchestra.run.smithi088.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-26T21:13:01.606 INFO:teuthology.orchestra.run.smithi088.stdout:(31/32): librbd1-19.0.0-2484.g42fc52b8.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-26T21:13:01.848 INFO:teuthology.orchestra.run.smithi088.stdout:(32/32): librados2-19.0.0-2484.g42fc52b8.el9.x8 5.4 MB/s | 3.4 MB 00:00 2024-03-26T21:13:01.853 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:01.854 INFO:teuthology.orchestra.run.smithi088.stdout:Total 15 MB/s | 59 MB 00:03 2024-03-26T21:13:01.883 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:13:01.917 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:13:01.918 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:13:02.403 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:13:02.404 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:13:03.823 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:13:03.909 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T21:13:03.936 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T21:13:03.971 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T21:13:03.993 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:13:04.009 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:13:04.152 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:13:04.207 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:13:04.584 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T21:13:04.631 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:13:04.706 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:13:04.792 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 8/34 2024-03-26T21:13:04.847 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:13:04.911 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:13:04.940 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T21:13:04.983 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 11/34 2024-03-26T21:13:05.052 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:13:05.076 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:13:05.123 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:13:05.158 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T21:13:05.267 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T21:13:05.301 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T21:13:05.341 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T21:13:05.382 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T21:13:05.411 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T21:13:05.444 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T21:13:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:13:05.602 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2024-03-26T21:13:05.603 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2024-03-26T21:13:05.603 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2024-03-26T21:13:05.603 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2024-03-26T21:13:05.603 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2024-03-26T21:13:05.603 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /root 2024-03-26T21:13:05.603 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2024-03-26T21:13:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:05.762 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T21:13:05.790 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:13:05.881 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:13:05.912 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 23/34 2024-03-26T21:13:05.974 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T21:13:06.435 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T21:13:06.460 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:13:06.514 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:13:06.534 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:13:06.558 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:13:06.559 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:06.559 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T21:13:06.559 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:13:06.559 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:13:06.559 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:06.560 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:13:06.582 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 27/34 2024-03-26T21:13:06.612 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 28/34 2024-03-26T21:13:06.646 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:13:06.647 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 6/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 8/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 9/34 2024-03-26T21:13:07.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 10/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 11/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 13/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T21:13:07.872 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T21:13:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/34 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T21:13:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:08.823 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:08.877 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:13:08.918 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:13:09.161 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-test 2024-03-26T21:13:09.466 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:13:09.504 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:13:09.588 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:13:09.588 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T21:13:09.588 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T21:13:09.588 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:09.613 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:13:09.721 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:28 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:09.875 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:09.876 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:09.876 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:13:09.876 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:09.876 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:13:09.876 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 48 M 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 163 k 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-03-26T21:13:09.877 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:09.878 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 49 M 2024-03-26T21:13:09.878 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 206 M 2024-03-26T21:13:09.878 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:13:10.121 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-26T21:13:10.180 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-26T21:13:10.221 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-26T21:13:10.330 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): libcephsqlite-19.0.0-2484.g42fc52b8.el9. 413 kB/s | 163 kB 00:00 2024-03-26T21:13:10.455 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 800 kB/s | 187 kB 00:00 2024-03-26T21:13:12.473 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): ceph-test-19.0.0-2484.g42fc52b8.el9.x86_ 19 MB/s | 48 MB 00:02 2024-03-26T21:13:12.475 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:12.476 INFO:teuthology.orchestra.run.smithi138.stdout:Total 19 MB/s | 49 MB 00:02 2024-03-26T21:13:12.491 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:13:12.513 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:13:12.513 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:13:12.905 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:13:12.906 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:13:13.471 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:13:13.567 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T21:13:13.599 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T21:13:13.639 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:13:13.682 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T21:13:13.710 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:13:18.251 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:13:18.327 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-03-26T21:13:19.121 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:20.071 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:13:20.095 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:13:20.095 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:20.095 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T21:13:20.096 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:13:20.096 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:13:20.096 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:20.098 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:13:20.166 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:13:20.166 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:13:20.908 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:13:20.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/6 2024-03-26T21:13:20.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/6 2024-03-26T21:13:20.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:13:20.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T21:13:20.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T21:13:21.280 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: jq-1.6-16.el9.x86_64 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:21.281 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:21.286 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 6/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 8/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 9/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 10/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 11/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 13/34 2024-03-26T21:13:21.287 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T21:13:21.288 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T21:13:21.289 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T21:13:21.289 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T21:13:21.289 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T21:13:21.289 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:13:21.289 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T21:13:21.289 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/34 2024-03-26T21:13:21.505 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph 2024-03-26T21:13:22.065 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:41 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:22.218 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 6.6 k 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 2.4 M 2024-03-26T21:13:22.221 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 1.5 M 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 249 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 4.7 M 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 17 M 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T21:13:22.222 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T21:13:22.223 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout:Install 36 Packages 2024-03-26T21:13:22.224 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:22.225 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 30 M 2024-03-26T21:13:22.225 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 108 M 2024-03-26T21:13:22.225 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:13:22.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:13:22.417 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:22.417 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-03-26T21:13:22.417 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.417 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.417 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T21:13:22.418 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T21:13:22.419 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T21:13:22.420 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:22.420 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:13:22.722 INFO:teuthology.orchestra.run.smithi138.stdout:(1/36): ceph-19.0.0-2484.g42fc52b8.el9.x86_64.r 32 kB/s | 6.6 kB 00:00 2024-03-26T21:13:22.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test 2024-03-26T21:13:23.197 INFO:teuthology.orchestra.run.smithi138.stdout:(2/36): ceph-mgr-19.0.0-2484.g42fc52b8.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-03-26T21:13:23.361 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:28 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:13:23.372 INFO:teuthology.orchestra.run.smithi138.stdout:(3/36): ceph-mds-19.0.0-2484.g42fc52b8.el9.x86_ 2.8 MB/s | 2.4 MB 00:00 2024-03-26T21:13:23.481 INFO:teuthology.orchestra.run.smithi138.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2484.g42fc 2.2 MB/s | 249 kB 00:00 2024-03-26T21:13:23.509 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:13:23.510 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:23.510 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 48 M 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 163 k 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:23.511 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-03-26T21:13:23.512 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:23.512 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 49 M 2024-03-26T21:13:23.512 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 206 M 2024-03-26T21:13:23.512 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:13:23.623 INFO:teuthology.orchestra.run.smithi138.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-26T21:13:23.674 INFO:teuthology.orchestra.run.smithi138.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-26T21:13:23.716 INFO:teuthology.orchestra.run.smithi138.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 754 kB/s | 32 kB 00:00 2024-03-26T21:13:23.758 INFO:teuthology.orchestra.run.smithi138.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-26T21:13:23.800 INFO:teuthology.orchestra.run.smithi138.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 924 kB/s | 39 kB 00:00 2024-03-26T21:13:23.818 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-26T21:13:23.850 INFO:teuthology.orchestra.run.smithi138.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T21:13:23.877 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-26T21:13:23.919 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-26T21:13:23.933 INFO:teuthology.orchestra.run.smithi138.stdout:(11/36): ceph-mon-19.0.0-2484.g42fc52b8.el9.x86 3.9 MB/s | 4.7 MB 00:01 2024-03-26T21:13:23.984 INFO:teuthology.orchestra.run.smithi138.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-03-26T21:13:24.009 INFO:teuthology.orchestra.run.smithi138.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-26T21:13:24.035 INFO:teuthology.orchestra.run.smithi138.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T21:13:24.060 INFO:teuthology.orchestra.run.smithi138.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-26T21:13:24.060 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-26T21:13:24.086 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): libcephsqlite-19.0.0-2484.g42fc52b8.el9. 357 kB/s | 163 kB 00:00 2024-03-26T21:13:24.144 INFO:teuthology.orchestra.run.smithi138.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-26T21:13:24.186 INFO:teuthology.orchestra.run.smithi138.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 829 kB/s | 125 kB 00:00 2024-03-26T21:13:24.445 INFO:teuthology.orchestra.run.smithi138.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 168 kB/s | 43 kB 00:00 2024-03-26T21:13:24.495 INFO:teuthology.orchestra.run.smithi138.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 431 kB/s | 151 kB 00:00 2024-03-26T21:13:24.620 INFO:teuthology.orchestra.run.smithi138.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 979 kB/s | 172 kB 00:00 2024-03-26T21:13:24.665 INFO:teuthology.orchestra.run.smithi138.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-26T21:13:24.688 INFO:teuthology.orchestra.run.smithi138.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 159 kB/s | 11 kB 00:00 2024-03-26T21:13:24.722 INFO:teuthology.orchestra.run.smithi138.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 307 kB/s | 18 kB 00:00 2024-03-26T21:13:24.755 INFO:teuthology.orchestra.run.smithi138.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 344 kB/s | 23 kB 00:00 2024-03-26T21:13:24.781 INFO:teuthology.orchestra.run.smithi138.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 331 kB/s | 19 kB 00:00 2024-03-26T21:13:24.823 INFO:teuthology.orchestra.run.smithi138.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-26T21:13:24.848 INFO:teuthology.orchestra.run.smithi138.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 690 kB/s | 46 kB 00:00 2024-03-26T21:13:24.915 INFO:teuthology.orchestra.run.smithi138.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 853 kB/s | 79 kB 00:00 2024-03-26T21:13:24.949 INFO:teuthology.orchestra.run.smithi138.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-26T21:13:24.991 INFO:teuthology.orchestra.run.smithi138.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2024-03-26T21:13:25.024 INFO:teuthology.orchestra.run.smithi138.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-26T21:13:25.066 INFO:teuthology.orchestra.run.smithi138.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-26T21:13:25.117 INFO:teuthology.orchestra.run.smithi138.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-26T21:13:25.184 INFO:teuthology.orchestra.run.smithi138.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2024-03-26T21:13:25.209 INFO:teuthology.orchestra.run.smithi138.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-26T21:13:26.193 INFO:teuthology.orchestra.run.smithi138.stdout:(36/36): ceph-osd-19.0.0-2484.g42fc52b8.el9.x86 5.6 MB/s | 17 MB 00:02 2024-03-26T21:13:26.198 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:26.199 INFO:teuthology.orchestra.run.smithi138.stdout:Total 7.5 MB/s | 30 MB 00:03 2024-03-26T21:13:26.223 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:13:26.266 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:13:26.266 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:13:26.679 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:13:26.680 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:13:27.984 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:13:28.045 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T21:13:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T21:13:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T21:13:28.185 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T21:13:28.231 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T21:13:28.264 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T21:13:28.319 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): ceph-test-19.0.0-2484.g42fc52b8.el9.x86_ 10 MB/s | 48 MB 00:04 2024-03-26T21:13:28.322 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:28.322 INFO:teuthology.orchestra.run.smithi088.stdout:Total 10 MB/s | 49 MB 00:04 2024-03-26T21:13:28.336 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:13:28.355 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T21:13:28.358 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:13:28.358 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:13:28.434 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T21:13:28.475 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T21:13:28.556 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T21:13:28.610 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T21:13:28.665 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T21:13:28.714 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T21:13:28.754 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:13:28.754 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:13:28.770 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T21:13:28.958 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T21:13:29.012 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T21:13:29.051 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T21:13:29.086 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T21:13:29.116 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T21:13:29.192 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T21:13:29.227 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:13:29.261 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T21:13:29.301 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T21:13:29.336 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T21:13:29.338 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:13:29.371 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T21:13:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T21:13:29.460 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T21:13:29.469 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T21:13:29.501 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:13:29.502 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T21:13:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T21:13:29.574 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T21:13:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:13:29.720 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T21:13:29.854 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:13:30.021 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 31/36 2024-03-26T21:13:30.051 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:13:30.075 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:13:30.075 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:30.075 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T21:13:30.076 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:13:30.076 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:13:30.076 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:31.510 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:13:31.537 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:13:31.537 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:31.537 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T21:13:31.537 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:13:31.537 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:13:31.537 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:31.978 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:13:32.001 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:13:32.001 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:32.001 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T21:13:32.001 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:13:32.001 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:13:32.001 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:32.229 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:13:32.251 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:13:32.252 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:32.252 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T21:13:32.252 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:13:32.252 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:13:32.252 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:32.317 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:13:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:13:34.182 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 6/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T21:13:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T21:13:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T21:13:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T21:13:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:13:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T21:13:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T21:13:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T21:13:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T21:13:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T21:13:34.188 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T21:13:35.267 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T21:13:35.268 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T21:13:35.269 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T21:13:35.270 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:35.270 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:35.579 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-base 2024-03-26T21:13:36.133 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:36.229 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:13:36.281 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:36.282 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:13:36.283 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:36.345 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install cephadm 2024-03-26T21:13:36.588 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:13:36.588 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/6 2024-03-26T21:13:36.588 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/6 2024-03-26T21:13:36.588 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:13:36.588 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T21:13:36.588 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T21:13:36.901 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:37.050 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:37.051 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:37.051 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:13:37.051 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:37.051 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 762 k 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 762 k 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 768 k 2024-03-26T21:13:37.052 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:13:37.061 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T21:13:37.061 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:37.061 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:13:37.061 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:37.061 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.6-16.el9.x86_64 2024-03-26T21:13:37.061 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:37.062 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T21:13:37.062 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T21:13:37.062 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T21:13:37.062 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:37.062 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:13:37.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph 2024-03-26T21:13:37.674 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm-19.0.0-2484.g42fc52b8.el9.noarch.rpm 1.2 MB/s | 762 kB 00:00 2024-03-26T21:13:37.675 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:37.675 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.2 MB/s | 762 kB 00:00 2024-03-26T21:13:37.675 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:13:37.680 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:13:37.680 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:13:37.686 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:13:37.686 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:13:37.762 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:13:37.894 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:42 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:13:38.046 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 6.6 k 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 2.4 M 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 1.5 M 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 249 k 2024-03-26T21:13:38.049 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 4.7 M 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 17 M 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T21:13:38.050 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T21:13:38.051 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout:Install 36 Packages 2024-03-26T21:13:38.052 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:38.053 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 30 M 2024-03-26T21:13:38.053 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 108 M 2024-03-26T21:13:38.053 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:13:38.158 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:38.239 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:38.617 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:38.654 INFO:teuthology.orchestra.run.smithi088.stdout:(1/36): ceph-19.0.0-2484.g42fc52b8.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-26T21:13:38.913 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:38.914 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:38.914 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:13:38.914 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:13:38.914 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:38.914 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:38.963 INFO:teuthology.orchestra.run.smithi088.stdout:(2/36): ceph-mgr-19.0.0-2484.g42fc52b8.el9.x86_ 3.1 MB/s | 1.5 MB 00:00 2024-03-26T21:13:39.115 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T21:13:39.346 INFO:teuthology.orchestra.run.smithi088.stdout:(3/36): ceph-mon-19.0.0-2484.g42fc52b8.el9.x86_ 6.8 MB/s | 4.7 MB 00:00 2024-03-26T21:13:39.405 INFO:teuthology.orchestra.run.smithi088.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2484.g42fc 4.2 MB/s | 249 kB 00:00 2024-03-26T21:13:39.555 INFO:teuthology.orchestra.run.smithi088.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-26T21:13:39.597 INFO:teuthology.orchestra.run.smithi088.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-26T21:13:39.664 INFO:teuthology.orchestra.run.smithi088.stdout:(7/36): ceph-mds-19.0.0-2484.g42fc52b8.el9.x86_ 2.0 MB/s | 2.4 MB 00:01 2024-03-26T21:13:39.689 INFO:teuthology.orchestra.run.smithi088.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 344 kB/s | 32 kB 00:00 2024-03-26T21:13:39.696 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:39.714 INFO:teuthology.orchestra.run.smithi088.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-03-26T21:13:39.739 INFO:teuthology.orchestra.run.smithi088.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-26T21:13:39.765 INFO:teuthology.orchestra.run.smithi088.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T21:13:39.824 INFO:teuthology.orchestra.run.smithi088.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-26T21:13:39.849 INFO:teuthology.orchestra.run.smithi088.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-26T21:13:39.855 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 145 k 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:13:39.856 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:13:39.857 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:13:39.857 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:39.857 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 145 k 2024-03-26T21:13:39.857 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 435 k 2024-03-26T21:13:39.857 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:13:39.874 INFO:teuthology.orchestra.run.smithi088.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T21:13:39.899 INFO:teuthology.orchestra.run.smithi088.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T21:13:39.983 INFO:teuthology.orchestra.run.smithi088.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-26T21:13:40.250 INFO:teuthology.orchestra.run.smithi088.stdout:(17/36): ceph-osd-19.0.0-2484.g42fc52b8.el9.x86 13 MB/s | 17 MB 00:01 2024-03-26T21:13:40.343 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-immutable-object-cache-19.0.0-2484.g42fc52 298 kB/s | 145 kB 00:00 2024-03-26T21:13:40.344 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:40.344 INFO:teuthology.orchestra.run.smithi138.stdout:Total 297 kB/s | 145 kB 00:00 2024-03-26T21:13:40.344 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:13:40.352 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:13:40.352 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:13:40.375 INFO:teuthology.orchestra.run.smithi088.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2024-03-26T21:13:40.401 INFO:teuthology.orchestra.run.smithi088.stdout:(19/36): python3-requests-2.25.1-8.el9.noarch.r 237 kB/s | 125 kB 00:00 2024-03-26T21:13:40.406 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:13:40.406 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:13:40.417 INFO:teuthology.orchestra.run.smithi088.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-03-26T21:13:40.501 INFO:teuthology.orchestra.run.smithi088.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-26T21:13:40.560 INFO:teuthology.orchestra.run.smithi088.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-26T21:13:40.561 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:13:40.585 INFO:teuthology.orchestra.run.smithi088.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-03-26T21:13:40.610 INFO:teuthology.orchestra.run.smithi088.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 732 kB/s | 172 kB 00:00 2024-03-26T21:13:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:13:40.636 INFO:teuthology.orchestra.run.smithi088.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-26T21:13:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:13:40.646 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:40.646 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T21:13:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:40.689 INFO:teuthology.orchestra.run.smithi088.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 188 kB/s | 19 kB 00:00 2024-03-26T21:13:40.711 INFO:teuthology.orchestra.run.smithi088.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-26T21:13:40.736 INFO:teuthology.orchestra.run.smithi088.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-26T21:13:40.786 INFO:teuthology.orchestra.run.smithi088.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-03-26T21:13:40.812 INFO:teuthology.orchestra.run.smithi088.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 641 kB/s | 79 kB 00:00 2024-03-26T21:13:40.845 INFO:teuthology.orchestra.run.smithi088.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-26T21:13:40.879 INFO:teuthology.orchestra.run.smithi088.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-26T21:13:40.913 INFO:teuthology.orchestra.run.smithi088.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-26T21:13:40.938 INFO:teuthology.orchestra.run.smithi088.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-26T21:13:40.971 INFO:teuthology.orchestra.run.smithi088.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-26T21:13:41.013 INFO:teuthology.orchestra.run.smithi088.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-26T21:13:41.016 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:41.016 INFO:teuthology.orchestra.run.smithi088.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-26T21:13:41.040 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:13:41.082 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:13:41.083 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:13:41.503 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:13:41.503 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:13:41.664 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:13:41.664 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:41.664 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:13:41.664 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:41.664 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:41.664 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:41.916 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr 2024-03-26T21:13:42.485 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:42.583 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:13:42.634 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:42.636 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:13:42.636 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:42.696 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T21:13:42.807 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:13:42.857 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T21:13:42.893 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T21:13:42.926 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T21:13:42.998 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T21:13:43.041 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T21:13:43.074 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T21:13:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T21:13:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T21:13:43.255 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T21:13:43.349 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T21:13:43.403 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T21:13:43.405 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:43.406 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-26T21:13:43.406 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:13:43.406 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-26T21:13:43.406 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 3.5 M 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 34 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 16 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:43.407 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:13:43.408 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-26T21:13:43.408 INFO:teuthology.orchestra.run.smithi138.stdout:Install 12 Packages 2024-03-26T21:13:43.408 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:43.408 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 8.6 M 2024-03-26T21:13:43.408 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 104 M 2024-03-26T21:13:43.408 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:13:43.441 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T21:13:43.490 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T21:13:43.550 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T21:13:43.684 INFO:teuthology.orchestra.run.smithi138.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2484.g42 180 kB/s | 34 kB 00:00 2024-03-26T21:13:43.743 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T21:13:43.776 INFO:teuthology.orchestra.run.smithi138.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2484.g42f 59 kB/s | 16 kB 00:00 2024-03-26T21:13:43.794 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T21:13:43.832 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T21:13:43.868 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T21:13:43.885 INFO:teuthology.orchestra.run.smithi138.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-26T21:13:43.898 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T21:13:43.910 INFO:teuthology.orchestra.run.smithi138.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-26T21:13:43.975 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T21:13:44.002 INFO:teuthology.orchestra.run.smithi138.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 7.2 MB/s | 863 kB 00:00 2024-03-26T21:13:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:13:44.050 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T21:13:44.090 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T21:13:44.111 INFO:teuthology.orchestra.run.smithi138.stdout:(6/12): ceph-mgr-dashboard-19.0.0-2484.g42fc52b 5.8 MB/s | 3.5 MB 00:00 2024-03-26T21:13:44.125 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T21:13:44.160 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T21:13:44.249 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T21:13:44.286 INFO:teuthology.orchestra.run.smithi138.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 68 kB/s | 19 kB 00:00 2024-03-26T21:13:44.290 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T21:13:44.363 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T21:13:44.509 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T21:13:44.545 INFO:teuthology.orchestra.run.smithi138.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 864 kB/s | 548 kB 00:00 2024-03-26T21:13:44.636 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:13:44.787 INFO:teuthology.orchestra.run.smithi138.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 289 kB/s | 144 kB 00:00 2024-03-26T21:13:44.799 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 31/36 2024-03-26T21:13:44.828 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:13:44.829 INFO:teuthology.orchestra.run.smithi138.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 108 kB/s | 31 kB 00:00 2024-03-26T21:13:44.854 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:13:44.854 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:44.854 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T21:13:44.854 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:13:44.854 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:13:44.854 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:44.912 INFO:teuthology.orchestra.run.smithi138.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.5 MB/s | 2.0 MB 00:00 2024-03-26T21:13:45.229 INFO:teuthology.orchestra.run.smithi138.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 426 kB/s | 188 kB 00:00 2024-03-26T21:13:45.234 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:45.234 INFO:teuthology.orchestra.run.smithi138.stdout:Total 4.7 MB/s | 8.6 MB 00:01 2024-03-26T21:13:45.254 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:13:45.273 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:13:45.273 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:13:45.457 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:13:45.458 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:13:45.982 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:13:46.071 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T21:13:46.115 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T21:13:46.162 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T21:13:46.201 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T21:13:46.298 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T21:13:46.313 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:13:46.339 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:13:46.339 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:46.339 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T21:13:46.339 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:13:46.339 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:13:46.340 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:46.428 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T21:13:46.521 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T21:13:46.583 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T21:13:46.611 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:13:46.652 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 10/12 2024-03-26T21:13:46.789 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:13:46.811 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:13:46.811 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:46.811 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T21:13:46.811 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:13:46.811 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:13:46.811 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:47.047 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:13:47.070 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:13:47.070 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:47.070 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T21:13:47.070 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:13:47.070 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:13:47.070 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:47.127 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:13:47.701 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 11/12 2024-03-26T21:13:47.745 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:13:48.965 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:13:48.965 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/36 2024-03-26T21:13:48.965 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/36 2024-03-26T21:13:48.965 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/36 2024-03-26T21:13:48.965 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/36 2024-03-26T21:13:48.965 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/36 2024-03-26T21:13:48.965 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 6/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T21:13:48.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T21:13:48.967 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T21:13:48.967 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:13:48.967 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T21:13:48.967 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T21:13:48.967 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T21:13:48.967 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T21:13:48.968 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T21:13:48.968 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T21:13:48.968 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T21:13:48.968 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T21:13:48.969 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:13:48.969 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T21:13:48.969 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T21:13:48.969 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T21:13:48.969 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T21:13:48.969 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 1/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 2/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 3/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T21:13:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:13:49.205 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T21:13:49.205 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T21:13:49.562 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T21:13:49.562 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:49.563 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:13:49.563 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T21:13:49.563 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:13:49.563 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:13:49.563 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:13:49.563 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T21:13:49.564 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T21:13:49.564 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T21:13:49.564 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T21:13:49.564 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T21:13:49.564 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T21:13:49.564 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T21:13:49.565 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T21:13:49.565 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:49.565 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:13:49.816 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T21:13:49.998 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T21:13:49.998 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T21:13:49.999 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T21:13:50.000 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T21:13:50.001 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T21:13:50.001 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T21:13:50.001 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T21:13:50.001 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T21:13:50.001 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T21:13:50.001 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:50.001 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:13:50.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base 2024-03-26T21:13:50.394 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:09 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:13:50.546 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 7.4 M 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T21:13:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout:Install 12 Packages 2024-03-26T21:13:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:13:50.549 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 37 M 2024-03-26T21:13:50.549 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 209 M 2024-03-26T21:13:50.549 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:13:50.941 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:13:51.036 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:13:51.088 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:13:51.089 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:13:51.089 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:13:51.126 INFO:teuthology.orchestra.run.smithi138.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2024-03-26T21:13:51.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install cephadm 2024-03-26T21:13:51.167 INFO:teuthology.orchestra.run.smithi138.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 425 kB/s | 17 kB 00:00 2024-03-26T21:13:51.235 INFO:teuthology.orchestra.run.smithi138.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-26T21:13:51.260 INFO:teuthology.orchestra.run.smithi138.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 380 kB/s | 34 kB 00:00 2024-03-26T21:13:51.443 INFO:teuthology.orchestra.run.smithi138.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-26T21:13:51.527 INFO:teuthology.orchestra.run.smithi138.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-26T21:13:51.552 INFO:teuthology.orchestra.run.smithi138.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-26T21:13:51.713 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:13:51.862 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 762 k 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 762 k 2024-03-26T21:13:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 768 k 2024-03-26T21:13:51.864 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:13:51.878 INFO:teuthology.orchestra.run.smithi138.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-26T21:13:52.012 INFO:teuthology.orchestra.run.smithi138.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-24 7.1 MB/s | 7.4 MB 00:01 2024-03-26T21:13:52.245 INFO:teuthology.orchestra.run.smithi138.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-26T21:13:52.287 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm-19.0.0-2484.g42fc52b8.el9.noarch.rpm 1.8 MB/s | 762 kB 00:00 2024-03-26T21:13:52.288 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:52.288 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.8 MB/s | 762 kB 00:00 2024-03-26T21:13:52.288 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:13:52.293 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:13:52.293 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:13:52.299 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:13:52.299 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:13:52.370 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:13:52.395 INFO:teuthology.orchestra.run.smithi138.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 954 kB/s | 803 kB 00:00 2024-03-26T21:13:52.463 INFO:teuthology.orchestra.run.smithi138.stdout:(12/12): libquadmath-11.4.1-3.el9.x86_64.rpm 334 kB/s | 195 kB 00:00 2024-03-26T21:13:52.464 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:52.465 INFO:teuthology.orchestra.run.smithi138.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-26T21:13:52.505 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:13:52.528 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:13:52.529 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:13:52.828 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:52.901 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:13:52.901 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:13:52.907 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:53.305 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:53.582 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:13:53.582 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:53.582 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:13:53.583 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:13:53.583 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:53.583 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:13:53.597 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:13:53.668 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T21:13:53.709 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T21:13:53.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T21:13:53.811 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:13:53.854 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T21:13:54.034 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:13:54.062 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:13:54.147 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T21:13:54.367 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:59 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:13:54.521 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:13:54.521 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:54.521 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-26T21:13:54.521 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:54.521 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:13:54.521 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 145 k 2024-03-26T21:13:54.522 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:54.522 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:13:54.522 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:13:54.522 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:13:54.522 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:54.522 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 145 k 2024-03-26T21:13:54.522 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 435 k 2024-03-26T21:13:54.523 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:13:54.743 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-19.0.0-2484.g42fc52 656 kB/s | 145 kB 00:00 2024-03-26T21:13:54.744 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:13:54.744 INFO:teuthology.orchestra.run.smithi088.stdout:Total 652 kB/s | 145 kB 00:00 2024-03-26T21:13:54.744 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:13:54.752 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:13:54.752 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:13:54.803 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:13:54.803 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:13:54.809 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T21:13:54.926 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T21:13:54.951 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:13:55.019 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:13:55.043 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:13:55.043 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:13:55.043 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T21:13:55.044 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:56.037 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:13:56.037 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:56.037 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:13:56.037 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:13:56.037 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:56.037 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:13:56.146 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T21:13:56.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr 2024-03-26T21:13:56.845 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:13:56.940 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:13:56.993 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:13:56.994 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:13:56.994 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:13:57.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T21:13:57.181 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T21:13:57.218 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:13:57.617 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:13:57.765 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:13:57.766 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-03-26T21:13:57.766 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:13:57.766 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 3.5 M 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 34 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 16 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T21:13:57.767 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 8.6 M 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 104 M 2024-03-26T21:13:57.768 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:13:58.397 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2484.g42f 81 kB/s | 16 kB 00:00 2024-03-26T21:13:58.480 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2484.g42 118 kB/s | 34 kB 00:00 2024-03-26T21:13:58.597 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-26T21:13:58.622 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-26T21:13:58.714 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 7.2 MB/s | 863 kB 00:00 2024-03-26T21:13:59.031 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 66 kB/s | 19 kB 00:00 2024-03-26T21:13:59.265 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 853 kB/s | 548 kB 00:00 2024-03-26T21:13:59.373 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-26T21:13:59.465 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 335 kB/s | 31 kB 00:00 2024-03-26T21:13:59.566 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.8 MB/s | 2.0 MB 00:00 2024-03-26T21:13:59.591 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-26T21:13:59.841 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:13:59.841 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 1/12 2024-03-26T21:13:59.841 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T21:13:59.841 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:13:59.841 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T21:13:59.841 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:13:59.842 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:13:59.842 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T21:13:59.842 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T21:13:59.842 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T21:13:59.842 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T21:13:59.842 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T21:14:00.324 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T21:14:00.325 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T21:14:00.325 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T21:14:00.325 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T21:14:00.325 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T21:14:00.325 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T21:14:00.325 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:00.325 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:00.658 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): ceph-mgr-dashboard-19.0.0-2484.g42fc52 1.4 MB/s | 3.5 MB 00:02 2024-03-26T21:14:00.663 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:00.664 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.0 MB/s | 8.6 MB 00:02 2024-03-26T21:14:00.684 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:00.703 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:00.703 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:00.780 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-rook 2024-03-26T21:14:00.887 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:01.341 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:20 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:01.411 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:01.483 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T21:14:01.493 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:01.494 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 49 k 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T21:14:01.495 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T21:14:01.496 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:01.497 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:01.497 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-26T21:14:01.497 INFO:teuthology.orchestra.run.smithi138.stdout:Install 13 Packages 2024-03-26T21:14:01.497 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:01.498 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 2.0 M 2024-03-26T21:14:01.498 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 26 M 2024-03-26T21:14:01.498 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:01.526 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T21:14:01.572 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T21:14:01.612 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T21:14:01.673 INFO:teuthology.orchestra.run.smithi138.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 186 kB/s | 24 kB 00:00 2024-03-26T21:14:01.698 INFO:teuthology.orchestra.run.smithi138.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-03-26T21:14:01.704 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T21:14:01.757 INFO:teuthology.orchestra.run.smithi138.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-26T21:14:01.782 INFO:teuthology.orchestra.run.smithi138.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-26T21:14:01.808 INFO:teuthology.orchestra.run.smithi138.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T21:14:01.829 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T21:14:01.833 INFO:teuthology.orchestra.run.smithi138.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 850 kB/s | 43 kB 00:00 2024-03-26T21:14:01.859 INFO:teuthology.orchestra.run.smithi138.stdout:(7/13): ceph-mgr-rook-19.0.0-2484.g42fc52b8.el9 156 kB/s | 49 kB 00:00 2024-03-26T21:14:01.914 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T21:14:01.959 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T21:14:01.995 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:14:02.027 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 10/12 2024-03-26T21:14:02.051 INFO:teuthology.orchestra.run.smithi138.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-03-26T21:14:02.076 INFO:teuthology.orchestra.run.smithi138.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-03-26T21:14:02.259 INFO:teuthology.orchestra.run.smithi138.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 506 kB/s | 203 kB 00:00 2024-03-26T21:14:02.293 INFO:teuthology.orchestra.run.smithi138.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2024-03-26T21:14:02.318 INFO:teuthology.orchestra.run.smithi138.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2024-03-26T21:14:02.343 INFO:teuthology.orchestra.run.smithi138.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-26T21:14:02.347 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:02.348 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-26T21:14:02.348 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:02.361 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:02.361 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:02.452 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:02.454 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:02.962 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:03.067 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 11/12 2024-03-26T21:14:03.103 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:14:03.130 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T21:14:03.195 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T21:14:03.258 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T21:14:03.292 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T21:14:03.329 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T21:14:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T21:14:03.540 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T21:14:03.598 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T21:14:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T21:14:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 1/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 2/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 3/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:14:04.399 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T21:14:04.400 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T21:14:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T21:14:04.470 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T21:14:04.493 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:14:04.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T21:14:04.738 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T21:14:04.739 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T21:14:04.740 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:04.740 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:05.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T21:14:05.310 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 1/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T21:14:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T21:14:05.580 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:10 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:05.685 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T21:14:05.686 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:05.687 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:05.741 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:05.742 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-26T21:14:05.742 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 7.4 M 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T21:14:05.743 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2024-03-26T21:14:05.744 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:05.745 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 37 M 2024-03-26T21:14:05.745 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 209 M 2024-03-26T21:14:05.745 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:05.924 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T21:14:06.182 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 171 kB/s | 32 kB 00:00 2024-03-26T21:14:06.224 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 416 kB/s | 17 kB 00:00 2024-03-26T21:14:06.266 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-26T21:14:06.325 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.1 MB/s | 3.0 MB 00:00 2024-03-26T21:14:06.482 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:25 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:06.509 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-26T21:14:06.567 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-26T21:14:06.592 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-26T21:14:06.631 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 141 k 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T21:14:06.632 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T21:14:06.633 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T21:14:06.633 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:06.633 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:06.633 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-26T21:14:06.633 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-03-26T21:14:06.633 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:06.633 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 6.8 M 2024-03-26T21:14:06.634 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 33 M 2024-03-26T21:14:06.634 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:06.801 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2024-03-26T21:14:06.893 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-24 8.3 MB/s | 7.4 MB 00:00 2024-03-26T21:14:07.010 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2024-03-26T21:14:07.052 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 781 kB/s | 195 kB 00:00 2024-03-26T21:14:07.175 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-26T21:14:07.177 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 865 kB/s | 245 kB 00:00 2024-03-26T21:14:07.179 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:07.179 INFO:teuthology.orchestra.run.smithi088.stdout:Total 26 MB/s | 37 MB 00:01 2024-03-26T21:14:07.219 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:07.243 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:07.243 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:07.301 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-26T21:14:07.326 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2484.g42fc52b8.e 395 kB/s | 141 kB 00:00 2024-03-26T21:14:07.451 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 408 kB/s | 51 kB 00:00 2024-03-26T21:14:07.485 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-26T21:14:07.536 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 247 kB/s | 58 kB 00:00 2024-03-26T21:14:07.537 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:07.538 INFO:teuthology.orchestra.run.smithi138.stdout:Total 7.5 MB/s | 6.8 MB 00:00 2024-03-26T21:14:07.550 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:07.560 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:07.560 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:07.599 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:07.599 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:07.650 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:07.650 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:07.946 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:08.119 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T21:14:08.188 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T21:14:08.299 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:08.366 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T21:14:08.397 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T21:14:08.499 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:14:08.542 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T21:14:08.708 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T21:14:08.720 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:14:08.750 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:14:08.791 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T21:14:08.826 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T21:14:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T21:14:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:14:09.473 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T21:14:09.595 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T21:14:09.690 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:14:09.691 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/6 2024-03-26T21:14:09.691 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T21:14:09.691 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T21:14:09.691 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T21:14:09.691 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T21:14:09.993 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T21:14:09.993 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:09.993 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:09.993 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:09.993 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T21:14:09.994 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T21:14:09.994 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T21:14:09.994 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T21:14:09.994 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T21:14:09.994 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:09.994 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:10.269 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-fuse 2024-03-26T21:14:10.816 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T21:14:10.828 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:10.986 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:10.987 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:10.987 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:14:10.987 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:10.987 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 890 k 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 969 k 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 2.8 M 2024-03-26T21:14:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:11.140 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 519 kB/s | 79 kB 00:00 2024-03-26T21:14:11.524 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-fuse-19.0.0-2484.g42fc52b8.el9.x86_ 1.6 MB/s | 890 kB 00:00 2024-03-26T21:14:11.525 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:11.526 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.8 MB/s | 969 kB 00:00 2024-03-26T21:14:11.526 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:11.546 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:11.547 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:11.623 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:11.624 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:11.803 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T21:14:11.874 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:14:11.962 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T21:14:12.041 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:14:12.778 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:14:12.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/2 2024-03-26T21:14:13.100 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T21:14:13.100 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:13.100 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:13.100 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T21:14:13.100 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:13.100 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:13.396 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-volume 2024-03-26T21:14:13.972 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:32 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:14.130 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:14.130 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-26T21:14:14.130 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:14.130 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 273 k 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 342 k 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.5 M 2024-03-26T21:14:14.131 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:14.275 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 479 kB/s | 69 kB 00:00 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 1/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T21:14:14.386 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T21:14:14.387 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T21:14:14.525 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-volume-19.0.0-2484.g42fc52b8.el9.no 694 kB/s | 273 kB 00:00 2024-03-26T21:14:14.526 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:14.526 INFO:teuthology.orchestra.run.smithi138.stdout:Total 864 kB/s | 342 kB 00:00 2024-03-26T21:14:14.526 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:14.535 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:14.535 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:14.574 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:14.574 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:14.719 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:14.856 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T21:14:14.856 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:14.856 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:14.856 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:14.856 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T21:14:14.856 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T21:14:14.856 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:14.857 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:14.911 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T21:14:14.953 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:14:14.969 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:14:14.969 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:14:14.969 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T21:14:14.969 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:15.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-rook 2024-03-26T21:14:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 1/2 2024-03-26T21:14:15.814 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:20 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:15.965 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:15.966 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-26T21:14:15.966 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:14:15.966 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 49 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T21:14:15.967 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2024-03-26T21:14:15.968 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:15.969 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 2.0 M 2024-03-26T21:14:15.969 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 26 M 2024-03-26T21:14:15.969 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T21:14:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:16.010 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T21:14:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:16.010 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:16.274 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 24 kB 00:00 2024-03-26T21:14:16.290 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-26T21:14:16.349 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-26T21:14:16.360 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados-devel 2024-03-26T21:14:16.374 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-26T21:14:16.399 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): ceph-mgr-rook-19.0.0-2484.g42fc52b8.el9 191 kB/s | 49 kB 00:00 2024-03-26T21:14:16.424 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-26T21:14:16.450 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 568 kB/s | 43 kB 00:00 2024-03-26T21:14:16.908 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 63 kB/s | 32 kB 00:00 2024-03-26T21:14:16.930 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:35 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:16.933 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 31 kB/s | 15 kB 00:00 2024-03-26T21:14:17.082 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:17.083 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:17.083 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-26T21:14:17.083 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 127 k 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 127 k 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 456 k 2024-03-26T21:14:17.084 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:17.117 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-26T21:14:17.276 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 245 kB/s | 203 kB 00:00 2024-03-26T21:14:17.301 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 488 kB/s | 90 kB 00:00 2024-03-26T21:14:17.393 INFO:teuthology.orchestra.run.smithi138.stdout:librados-devel-19.0.0-2484.g42fc52b8.el9.x86_64 410 kB/s | 127 kB 00:00 2024-03-26T21:14:17.394 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:17.394 INFO:teuthology.orchestra.run.smithi138.stdout:Total 408 kB/s | 127 kB 00:00 2024-03-26T21:14:17.394 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:17.400 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:17.400 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:17.456 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:17.456 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:17.742 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:18.231 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:18.334 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 733 kB/s | 1.0 MB 00:01 2024-03-26T21:14:18.339 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:18.339 INFO:teuthology.orchestra.run.smithi088.stdout:Total 883 kB/s | 2.0 MB 00:02 2024-03-26T21:14:18.340 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:18.354 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:18.355 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:18.450 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:18.451 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:18.483 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:18.483 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:18.483 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:18.484 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:18.484 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:18.484 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:18.739 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs2 2024-03-26T21:14:18.970 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:19.148 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T21:14:19.201 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T21:14:19.265 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T21:14:19.303 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T21:14:19.308 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:19.344 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T21:14:19.410 INFO:teuthology.orchestra.run.smithi138.stdout:Package libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:19.438 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T21:14:19.464 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:19.465 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:14:19.465 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:19.523 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs-devel 2024-03-26T21:14:19.546 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T21:14:19.596 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T21:14:20.083 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:20.232 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:20.233 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:20.233 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-26T21:14:20.233 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 32 k 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 32 k 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 142 k 2024-03-26T21:14:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:20.321 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T21:14:20.367 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T21:14:20.409 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-devel-19.0.0-2484.g42fc52b8.el9.x86_6 182 kB/s | 32 kB 00:00 2024-03-26T21:14:20.409 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:20.410 INFO:teuthology.orchestra.run.smithi138.stdout:Total 181 kB/s | 32 kB 00:00 2024-03-26T21:14:20.410 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:20.413 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:20.413 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:20.422 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T21:14:20.444 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:20.445 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:20.474 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T21:14:20.496 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:14:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:20.665 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:21.034 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:21.290 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:21.290 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:21.290 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:21.290 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:21.290 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:21.290 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 1/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T21:14:21.303 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T21:14:21.450 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados2 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T21:14:21.653 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T21:14:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T21:14:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T21:14:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T21:14:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T21:14:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T21:14:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:21.654 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:21.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T21:14:22.025 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:22.124 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:22.176 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:22.177 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:14:22.177 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:22.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librbd1 2024-03-26T21:14:22.454 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 141 k 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T21:14:22.604 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T21:14:22.605 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-26T21:14:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-03-26T21:14:22.605 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 6.8 M 2024-03-26T21:14:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 33 M 2024-03-26T21:14:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:22.798 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:22.897 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:22.951 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:22.952 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:14:22.952 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:23.010 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rados 2024-03-26T21:14:23.037 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-26T21:14:23.187 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-26T21:14:23.279 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2484.g42fc52b8.e 345 kB/s | 141 kB 00:00 2024-03-26T21:14:23.429 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 240 kB/s | 58 kB 00:00 2024-03-26T21:14:23.463 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-26T21:14:23.513 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 218 kB/s | 51 kB 00:00 2024-03-26T21:14:23.517 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:23.517 INFO:teuthology.orchestra.run.smithi088.stdout:Total 7.5 MB/s | 6.8 MB 00:00 2024-03-26T21:14:23.529 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:23.539 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:23.539 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:23.571 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:23.627 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:23.628 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:23.668 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:23.720 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:23.721 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:14:23.721 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:23.780 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rgw 2024-03-26T21:14:23.922 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:24.078 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T21:14:24.123 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T21:14:24.344 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:24.442 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:24.493 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:24.494 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:14:24.494 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:24.555 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-cephfs 2024-03-26T21:14:24.647 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T21:14:24.742 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T21:14:24.804 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T21:14:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:14:25.111 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:25.207 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:25.260 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:25.261 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:14:25.261 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:25.334 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rbd 2024-03-26T21:14:25.620 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:14:25.620 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/6 2024-03-26T21:14:25.620 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T21:14:25.620 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T21:14:25.620 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T21:14:25.620 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T21:14:25.897 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:25.927 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T21:14:25.927 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:25.927 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:25.928 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T21:14:25.928 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T21:14:25.928 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T21:14:25.928 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T21:14:25.928 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T21:14:25.928 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:25.994 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:26.045 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:26.046 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-26T21:14:26.046 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:26.109 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-fuse 2024-03-26T21:14:26.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse 2024-03-26T21:14:26.666 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:26.739 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:26.815 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:26.815 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:26.815 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-26T21:14:26.815 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:26.815 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:26.816 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 86 k 2024-03-26T21:14:26.816 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:26.816 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:26.816 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:26.816 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:14:26.816 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:26.816 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 86 k 2024-03-26T21:14:26.817 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 231 k 2024-03-26T21:14:26.817 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:26.889 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 890 k 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-03-26T21:14:26.891 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:26.891 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 969 k 2024-03-26T21:14:26.891 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.8 M 2024-03-26T21:14:26.891 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:27.022 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 598 kB/s | 79 kB 00:00 2024-03-26T21:14:27.037 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 389 kB/s | 86 kB 00:00 2024-03-26T21:14:27.038 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:27.038 INFO:teuthology.orchestra.run.smithi138.stdout:Total 387 kB/s | 86 kB 00:00 2024-03-26T21:14:27.038 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:27.045 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:27.045 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:27.101 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:27.101 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:27.223 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-fuse-19.0.0-2484.g42fc52b8.el9.x86_ 2.6 MB/s | 890 kB 00:00 2024-03-26T21:14:27.224 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:27.224 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.8 MB/s | 969 kB 00:00 2024-03-26T21:14:27.224 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:27.244 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:27.244 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:27.248 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:27.318 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:27.319 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:27.352 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:27.663 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T21:14:27.731 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:14:27.748 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:27.999 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:27.999 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:27.999 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:27.999 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:27.999 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:28.000 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:28.274 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-mirror 2024-03-26T21:14:28.503 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:14:28.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/2 2024-03-26T21:14:28.810 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T21:14:28.810 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:28.811 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:28.811 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T21:14:28.811 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:28.811 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:28.843 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:47 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:28.999 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:29.000 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:29.000 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:14:29.001 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:29.002 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.1 M 2024-03-26T21:14:29.002 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 13 M 2024-03-26T21:14:29.002 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:29.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-volume 2024-03-26T21:14:29.663 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:34 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:29.813 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 273 k 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:29.814 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:29.815 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:29.815 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-03-26T21:14:29.815 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:29.815 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 342 k 2024-03-26T21:14:29.815 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.5 M 2024-03-26T21:14:29.815 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:29.952 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 502 kB/s | 69 kB 00:00 2024-03-26T21:14:30.145 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-volume-19.0.0-2484.g42fc52b8.el9.no 831 kB/s | 273 kB 00:00 2024-03-26T21:14:30.146 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:30.146 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.0 MB/s | 342 kB 00:00 2024-03-26T21:14:30.146 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:30.154 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:30.155 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:30.188 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:30.188 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:30.339 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:30.468 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-mirror-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 2.1 MB/s | 3.1 MB 00:01 2024-03-26T21:14:30.469 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:30.469 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.1 MB/s | 3.1 MB 00:01 2024-03-26T21:14:30.469 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:30.480 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:30.480 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:30.522 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T21:14:30.555 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:14:30.556 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:30.556 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:30.571 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:14:30.571 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:14:30.571 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T21:14:30.571 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:30.980 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:31.045 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:31.068 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:31.068 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:14:31.068 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T21:14:31.068 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:14:31.068 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:14:31.068 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:31.342 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 1/2 2024-03-26T21:14:31.664 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T21:14:31.664 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:31.664 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:31.664 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:14:31.664 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T21:14:31.664 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:31.664 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:31.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel 2024-03-26T21:14:32.230 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:32.231 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:32.231 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:32.231 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:32.231 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:32.231 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:32.457 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-nbd 2024-03-26T21:14:32.548 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:32.698 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:32.698 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 127 k 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 127 k 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 456 k 2024-03-26T21:14:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:33.026 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:52 ago on Tue 26 Mar 2024 09:12:41 PM UTC. 2024-03-26T21:14:33.050 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-19.0.0-2484.g42fc52b8.el9.x86_64 362 kB/s | 127 kB 00:00 2024-03-26T21:14:33.050 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:33.051 INFO:teuthology.orchestra.run.smithi088.stdout:Total 360 kB/s | 127 kB 00:00 2024-03-26T21:14:33.051 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:33.056 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:33.056 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:33.113 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:33.113 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:33.176 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-26T21:14:33.177 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:33.177 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-26T21:14:33.177 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:33.177 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 171 k 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 171 k 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 494 k 2024-03-26T21:14:33.178 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-26T21:14:33.316 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:33.423 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:33.526 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-nbd-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 493 kB/s | 171 kB 00:00 2024-03-26T21:14:33.526 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:33.526 INFO:teuthology.orchestra.run.smithi138.stdout:Total 490 kB/s | 171 kB 00:00 2024-03-26T21:14:33.527 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-26T21:14:33.534 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-26T21:14:33.534 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-26T21:14:33.589 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-26T21:14:33.590 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-26T21:14:33.739 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-26T21:14:33.841 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:33.896 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:34.191 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:34.191 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:34.191 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:34.192 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:34.192 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:34.192 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:34.399 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:34.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2 2024-03-26T21:14:34.688 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:34.688 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:34.689 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-26T21:14:34.689 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:34.689 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:14:34.689 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-26T21:14:34.921 DEBUG:teuthology.parallel:result is None 2024-03-26T21:14:34.985 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:35.087 INFO:teuthology.orchestra.run.smithi088.stdout:Package libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:35.138 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:35.139 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:14:35.139 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:35.208 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs-devel 2024-03-26T21:14:35.768 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:35.916 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:35.916 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:35.916 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-26T21:14:35.916 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:35.916 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:35.916 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 32 k 2024-03-26T21:14:35.917 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:35.917 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:35.917 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:35.917 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:14:35.917 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:35.917 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 32 k 2024-03-26T21:14:35.917 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 142 k 2024-03-26T21:14:35.918 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:36.153 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs-devel-19.0.0-2484.g42fc52b8.el9.x86_6 134 kB/s | 32 kB 00:00 2024-03-26T21:14:36.154 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:36.154 INFO:teuthology.orchestra.run.smithi088.stdout:Total 134 kB/s | 32 kB 00:00 2024-03-26T21:14:36.154 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:36.158 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:36.158 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:36.180 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:36.180 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:36.277 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:36.385 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:36.776 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:37.043 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:37.043 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:37.043 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:37.043 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:37.043 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:37.043 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:37.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2 2024-03-26T21:14:37.808 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:37.904 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:37.955 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:37.956 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:14:37.956 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:38.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1 2024-03-26T21:14:38.570 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:38.667 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:38.720 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:38.721 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:14:38.721 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:38.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados 2024-03-26T21:14:39.339 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:39.436 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:39.490 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:39.491 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:14:39.491 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:39.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw 2024-03-26T21:14:40.102 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:40.199 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:40.250 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:40.251 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:14:40.251 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:40.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs 2024-03-26T21:14:40.863 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:40.960 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:41.010 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:41.011 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:14:41.011 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:41.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd 2024-03-26T21:14:41.630 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:41.729 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:14:41.780 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:41.781 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-26T21:14:41.781 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:41.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse 2024-03-26T21:14:42.402 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:47 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:42.555 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 86 k 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:42.556 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 86 k 2024-03-26T21:14:42.557 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 231 k 2024-03-26T21:14:42.557 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:42.890 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 258 kB/s | 86 kB 00:00 2024-03-26T21:14:42.890 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:42.891 INFO:teuthology.orchestra.run.smithi088.stdout:Total 257 kB/s | 86 kB 00:00 2024-03-26T21:14:42.891 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:42.898 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:42.898 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:42.954 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:42.954 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:43.101 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:43.213 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:43.641 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:43.940 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:43.940 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:43.940 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:43.940 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:43.940 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:43.940 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:44.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror 2024-03-26T21:14:44.743 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:44.897 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:44.897 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:44.897 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-26T21:14:44.897 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:44.897 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:44.897 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:14:44.898 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:44.898 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:44.898 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:44.898 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:14:44.898 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:44.898 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.1 M 2024-03-26T21:14:44.899 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 13 M 2024-03-26T21:14:44.899 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:45.867 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 3.2 MB/s | 3.1 MB 00:00 2024-03-26T21:14:45.868 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:45.868 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.2 MB/s | 3.1 MB 00:00 2024-03-26T21:14:45.868 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:45.878 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:45.878 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:45.951 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:45.951 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:46.369 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:46.435 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:46.459 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:46.460 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:14:46.460 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T21:14:46.460 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:14:46.460 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:14:46.460 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:47.598 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:47.598 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:47.598 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:47.598 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:47.599 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:47.599 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:47.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd 2024-03-26T21:14:48.402 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:53 ago on Tue 26 Mar 2024 09:12:55 PM UTC. 2024-03-26T21:14:48.551 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 171 k 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 171 k 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 494 k 2024-03-26T21:14:48.552 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-26T21:14:48.912 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 475 kB/s | 171 kB 00:00 2024-03-26T21:14:48.913 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:14:48.913 INFO:teuthology.orchestra.run.smithi088.stdout:Total 474 kB/s | 171 kB 00:00 2024-03-26T21:14:48.913 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-26T21:14:48.920 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-26T21:14:48.920 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-26T21:14:48.976 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-26T21:14:48.976 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-26T21:14:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-26T21:14:49.219 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:49.671 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:49.937 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:14:49.938 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:49.938 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-26T21:14:49.938 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:14:49.938 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:14:49.938 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-26T21:14:50.165 DEBUG:teuthology.parallel:result is None 2024-03-26T21:14:50.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:14:50.310 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T21:14:50.333 INFO:teuthology.orchestra.run.smithi088.stdout:19.0.0-2484.g42fc52b8.el9 2024-03-26T21:14:50.334 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2484.g42fc52b8.el9 2024-03-26T21:14:50.334 INFO:teuthology.task.install:The correct ceph version 19.0.0-2484.g42fc52b8 is installed. 2024-03-26T21:14:50.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:14:50.477 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T21:14:50.502 INFO:teuthology.orchestra.run.smithi138.stdout:19.0.0-2484.g42fc52b8.el9 2024-03-26T21:14:50.502 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2484.g42fc52b8.el9 2024-03-26T21:14:50.503 INFO:teuthology.task.install:The correct ceph version 19.0.0-2484.g42fc52b8 is installed. 2024-03-26T21:14:50.504 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T21:14:50.505 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:14:50.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T21:14:50.532 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:14:50.532 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T21:14:50.574 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T21:14:50.575 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:14:50.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T21:14:50.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T21:14:50.668 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:14:50.668 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T21:14:50.698 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T21:14:50.767 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T21:14:50.767 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:14:50.767 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T21:14:50.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T21:14:50.861 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:14:50.861 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T21:14:50.890 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T21:14:50.960 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T21:14:50.961 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:14:50.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T21:14:50.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T21:14:51.054 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:14:51.054 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T21:14:51.082 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T21:14:51.151 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T21:14:51.255 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'mon_bind_msgr2': False, 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'cephadm_mode': 'cephadm-package'} 2024-03-26T21:14:51.255 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:14:51.255 INFO:tasks.cephadm:Cluster fsid is e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:14:51.256 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T21:14:51.256 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.88:6789]', 'mon.c': '[v1:172.21.15.88:6790]', 'mon.b': '[v1:172.21.15.138:6789]'} 2024-03-26T21:14:51.256 INFO:tasks.cephadm:First mon is mon.a on smithi088 2024-03-26T21:14:51.256 INFO:tasks.cephadm:First mgr is y 2024-03-26T21:14:51.256 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T21:14:51.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2024-03-26T21:14:51.284 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname $(hostname -s) 2024-03-26T21:14:51.322 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 on all hosts... 2024-03-26T21:14:51.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 pull 2024-03-26T21:14:51.326 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 pull 2024-03-26T21:14:52.131 INFO:teuthology.orchestra.run.smithi088.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:14:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:15:33.826 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-03-26T21:15:33.826 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_version": "ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev)", 2024-03-26T21:15:33.826 INFO:teuthology.orchestra.run.smithi138.stdout: "image_id": "a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063", 2024-03-26T21:15:33.826 INFO:teuthology.orchestra.run.smithi138.stdout: "repo_digests": [ 2024-03-26T21:15:33.826 INFO:teuthology.orchestra.run.smithi138.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d" 2024-03-26T21:15:33.827 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-03-26T21:15:33.827 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-03-26T21:15:37.316 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-03-26T21:15:37.316 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev)", 2024-03-26T21:15:37.316 INFO:teuthology.orchestra.run.smithi088.stdout: "image_id": "a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063", 2024-03-26T21:15:37.316 INFO:teuthology.orchestra.run.smithi088.stdout: "repo_digests": [ 2024-03-26T21:15:37.316 INFO:teuthology.orchestra.run.smithi088.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d" 2024-03-26T21:15:37.317 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-03-26T21:15:37.317 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-03-26T21:15:37.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2024-03-26T21:15:37.380 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph 2024-03-26T21:15:37.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2024-03-26T21:15:37.447 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /etc/ceph 2024-03-26T21:15:37.487 INFO:tasks.cephadm:Writing seed config... 2024-03-26T21:15:37.489 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T21:15:37.489 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T21:15:37.489 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T21:15:37.489 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T21:15:37.490 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-26T21:15:37.491 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-26T21:15:37.491 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T21:15:37.492 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:15:37.492 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T21:15:37.507 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e240a1a4-ebb5-11ee-95d0-87774f69a715 mon election default strategy = 1 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-26T21:15:37.507 DEBUG:teuthology.orchestra.run.smithi088:mon.a> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T21:15:37.549 DEBUG:teuthology.orchestra.run.smithi088:mgr.y> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T21:15:37.593 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T21:15:37.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 -v bootstrap --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.88:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:15:37.783 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:15:37.784 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55', '-v', 'bootstrap', '--fsid', 'e240a1a4-ebb5-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.88:6789]', '--skip-admin-label'] 2024-03-26T21:15:37.784 INFO:teuthology.orchestra.run.smithi088.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T21:15:37.784 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying podman|docker is present... 2024-03-26T21:15:37.809 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T21:15:37.809 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying lvm2 is present... 2024-03-26T21:15:37.809 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying time synchronization is in place... 2024-03-26T21:15:37.815 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T21:15:37.815 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T21:15:37.821 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T21:15:37.821 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2024-03-26T21:15:37.827 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout enabled 2024-03-26T21:15:37.834 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout active 2024-03-26T21:15:37.834 INFO:teuthology.orchestra.run.smithi088.stdout:Unit chronyd.service is enabled and running 2024-03-26T21:15:37.834 INFO:teuthology.orchestra.run.smithi088.stdout:Repeating the final host check... 2024-03-26T21:15:37.856 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T21:15:37.857 INFO:teuthology.orchestra.run.smithi088.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T21:15:37.857 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl is present 2024-03-26T21:15:37.857 INFO:teuthology.orchestra.run.smithi088.stdout:lvcreate is present 2024-03-26T21:15:37.862 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T21:15:37.862 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T21:15:37.868 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T21:15:37.868 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2024-03-26T21:15:37.874 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout enabled 2024-03-26T21:15:37.880 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout active 2024-03-26T21:15:37.880 INFO:teuthology.orchestra.run.smithi088.stdout:Unit chronyd.service is enabled and running 2024-03-26T21:15:37.880 INFO:teuthology.orchestra.run.smithi088.stdout:Host looks OK 2024-03-26T21:15:37.881 INFO:teuthology.orchestra.run.smithi088.stdout:Cluster fsid: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:37.881 INFO:teuthology.orchestra.run.smithi088.stdout:Acquiring lock 140328195883696 on /run/cephadm/e240a1a4-ebb5-11ee-95d0-87774f69a715.lock 2024-03-26T21:15:37.881 INFO:teuthology.orchestra.run.smithi088.stdout:Lock 140328195883696 acquired on /run/cephadm/e240a1a4-ebb5-11ee-95d0-87774f69a715.lock 2024-03-26T21:15:37.881 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying IP 172.21.15.88 port 6789 ... 2024-03-26T21:15:37.881 INFO:teuthology.orchestra.run.smithi088.stdout:Base mon IP(s) is [172.21.15.88:6789], mon addrv is [v1:172.21.15.88:6789] 2024-03-26T21:15:37.884 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.88 metric 100 2024-03-26T21:15:37.884 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.88 metric 100 2024-03-26T21:15:37.886 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T21:15:37.886 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T21:15:37.886 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1783sec hoplimit 64 pref medium 2024-03-26T21:15:37.889 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T21:15:37.889 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T21:15:37.889 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T21:15:37.889 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T21:15:37.889 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link 2024-03-26T21:15:37.889 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T21:15:37.890 INFO:teuthology.orchestra.run.smithi088.stdout:Mon IP `172.21.15.88` is in CIDR network `172.21.0.0/20` 2024-03-26T21:15:37.890 INFO:teuthology.orchestra.run.smithi088.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-26T21:15:37.893 INFO:teuthology.orchestra.run.smithi088.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T21:15:37.895 INFO:teuthology.orchestra.run.smithi088.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:15:38.426 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stdout a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 2024-03-26T21:15:38.427 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:15:38.427 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T21:15:38.427 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Copying blob sha256:2e177a2f3361febef0925bf17c6fa96e6cd4af765a1dda3326a2b328e0c37fe9 2024-03-26T21:15:38.427 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-26T21:15:38.427 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Copying config sha256:a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 2024-03-26T21:15:38.427 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T21:15:39.457 INFO:teuthology.orchestra.run.smithi088.stdout:ceph: stdout ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev) 2024-03-26T21:15:39.457 INFO:teuthology.orchestra.run.smithi088.stdout:Ceph version: ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev) 2024-03-26T21:15:39.457 INFO:teuthology.orchestra.run.smithi088.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T21:15:40.480 INFO:teuthology.orchestra.run.smithi088.stdout:stat: stdout 167 167 2024-03-26T21:15:40.480 INFO:teuthology.orchestra.run.smithi088.stdout:Creating initial keys... 2024-03-26T21:15:41.586 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph-authtool: stdout AQD8OgNmeaiHMxAA+rJv3EYJzip3g1ol+0nOCw== 2024-03-26T21:15:42.532 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph-authtool: stdout AQD9OgNm6PfhOhAAX2FHdVbfGJOQnVKcPj75BQ== 2024-03-26T21:15:43.464 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph-authtool: stdout AQD+OgNmrqxANxAAdO4BWKXMjev+iJXEsGfgkw== 2024-03-26T21:15:43.464 INFO:teuthology.orchestra.run.smithi088.stdout:Creating initial monmap... 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool for a [v1:172.21.15.88:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = quincy 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: set fsid to e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:15:44.412 INFO:teuthology.orchestra.run.smithi088.stdout:Creating mon... 2024-03-26T21:15:45.343 INFO:teuthology.orchestra.run.smithi088.stdout:create mon.a on 2024-03-26T21:15:45.552 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T21:15:45.691 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T21:15:45.859 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715.target. 2024-03-26T21:15:45.860 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715.target. 2024-03-26T21:15:46.087 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a 2024-03-26T21:15:46.088 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T21:15:46.250 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715.target.wants/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@.service. 2024-03-26T21:15:46.534 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 podman[29561]: 2024-03-26 21:15:46.39572208 +0000 UTC m=+0.017942036 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:15:46.534 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 podman[29561]: 2024-03-26 21:15:46.540913186 +0000 UTC m=+0.163133136 container create aadec77452d443da5a811432ed657797aa6948b8556cb5eb7e681397d820e8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326) 2024-03-26T21:15:46.820 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 podman[29561]: 2024-03-26 21:15:46.718286658 +0000 UTC m=+0.340506609 container init aadec77452d443da5a811432ed657797aa6948b8556cb5eb7e681397d820e8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T21:15:46.820 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 podman[29561]: 2024-03-26 21:15:46.723385244 +0000 UTC m=+0.345605195 container start aadec77452d443da5a811432ed657797aa6948b8556cb5eb7e681397d820e8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:15:46.820 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 ceph-mon[29599]: mkfs e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:46.820 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 ceph-mon[29599]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T21:15:46.820 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 bash[29561]: aadec77452d443da5a811432ed657797aa6948b8556cb5eb7e681397d820e8d4 2024-03-26T21:15:46.832 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:15:46.832 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2024-03-26T21:15:46.839 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:15:46.839 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2024-03-26T21:15:46.839 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2024-03-26T21:15:46.839 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T21:15:46.840 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mon to start... 2024-03-26T21:15:46.840 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mon... 2024-03-26T21:15:47.073 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:46 smithi088 systemd[1]: Started Ceph mon.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:15:47.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T21:15:47.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout id: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:47.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T21:15:47.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:15:47.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout services: 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.6061s) 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout data: 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.313+0000 7f8e9d6e1700 1 Processor -- start 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.313+0000 7f8e9d6e1700 1 -- start start 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e9d6e1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e9807a6f0 con 0x7f8e98104d50 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e96ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8e98104d50 0x7f8e98105170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50508/0 (socket says 172.21.15.88:50508) 2024-03-26T21:15:47.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e96ffd700 1 -- 172.21.15.88:0/2185609480 learned_addr learned my addr 172.21.15.88:0/2185609480 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e95ffb700 1 -- 172.21.15.88:0/2185609480 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319378696 0 0) 0x7f8e9807a6f0 con 0x7f8e98104d50 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e95ffb700 1 -- 172.21.15.88:0/2185609480 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e840035b0 con 0x7f8e98104d50 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e95ffb700 1 -- 172.21.15.88:0/2185609480 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3324420247 0 0) 0x7f8e840035b0 con 0x7f8e98104d50 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e95ffb700 1 -- 172.21.15.88:0/2185609480 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e98106890 con 0x7f8e98104d50 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.314+0000 7f8e95ffb700 1 -- 172.21.15.88:0/2185609480 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f8e90001900 con 0x7f8e98104d50 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.315+0000 7f8e95ffb700 1 -- 172.21.15.88:0/2185609480 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e90001ca0 con 0x7f8e98104d50 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.315+0000 7f8e95ffb700 1 -- 172.21.15.88:0/2185609480 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f8e90002020 con 0x7f8e98104d50 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.315+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2185609480 >> v1:172.21.15.88:6789/0 conn(0x7f8e98104d50 legacy=0x7f8e98105170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.315+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2185609480 shutdown_connections 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.315+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2185609480 >> 172.21.15.88:0/2185609480 conn(0x7f8e981008f0 msgr2=0x7f8e98102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.315+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2185609480 shutdown_connections 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.315+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2185609480 wait complete. 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e9d6e1700 1 Processor -- start 2024-03-26T21:15:47.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e9d6e1700 1 -- start start 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e9d6e1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e9819c1a0 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e96ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8e98104d50 0x7f8e9819ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50518/0 (socket says 172.21.15.88:50518) 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e96ffd700 1 -- 172.21.15.88:0/2182402790 learned_addr learned my addr 172.21.15.88:0/2182402790 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064307773 0 0) 0x7f8e9819c1a0 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e840031f0 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.316+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 913104860 0 0) 0x7f8e840031f0 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e9819c1a0 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f8e90001800 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3490001459 0 0) 0x7f8e9819c1a0 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9819c370 con 0x7f8e98104d50 2024-03-26T21:15:47.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9819c660 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e981a0160 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e90005050 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f8e90001ca0 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8e90005650 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.317+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7f8e90006ec0 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.318+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e78005150 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.320+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8e90005b20 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.390+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "status"} v 0) -- 0x7f8e78005c50 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.390+0000 7f8e8ffff700 1 -- 172.21.15.88:0/2182402790 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+315 (unknown 1155462804 0 422465192) 0x7f8e90005d80 con 0x7f8e98104d50 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.391+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 >> v1:172.21.15.88:6789/0 conn(0x7f8e98104d50 legacy=0x7f8e9819ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.391+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 shutdown_connections 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.391+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 >> 172.21.15.88:0/2182402790 conn(0x7f8e981008f0 msgr2=0x7f8e98192520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.391+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 shutdown_connections 2024-03-26T21:15:47.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:47.391+0000 7f8e9d6e1700 1 -- 172.21.15.88:0/2182402790 wait complete. 2024-03-26T21:15:47.975 INFO:teuthology.orchestra.run.smithi088.stdout:mon is available 2024-03-26T21:15:47.975 INFO:teuthology.orchestra.run.smithi088.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T21:15:48.089 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T21:15:48.089 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: monmap epoch 1 2024-03-26T21:15:48.089 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:48.089 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: last_changed 2024-03-26T21:15:43.819826+0000 2024-03-26T21:15:48.089 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:15:48.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: min_mon_release 19 (squid) 2024-03-26T21:15:48.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: election_strategy: 1 2024-03-26T21:15:48.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:15:48.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: fsmap 2024-03-26T21:15:48.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T21:15:48.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: mgrmap e1: no daemons active 2024-03-26T21:15:48.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:47 smithi088 ceph-mon[29599]: from='client.? v1:172.21.15.88:0/2182402790' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T21:15:49.058 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:48 smithi088 ceph-mon[29599]: from='client.? v1:172.21.15.88:0/2489535387' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T21:15:49.059 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:48 smithi088 ceph-mon[29599]: from='client.? v1:172.21.15.88:0/2489535387' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout fsid = e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.88:6789] 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T21:15:49.071 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.436+0000 7f36d26f2700 1 Processor -- start 2024-03-26T21:15:49.072 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.436+0000 7f36d26f2700 1 -- start start 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.436+0000 7f36d26f2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f36cc108f10 con 0x7f36cc104ae0 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36cbfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f36cc104ae0 0x7f36cc104f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50520/0 (socket says 172.21.15.88:50520) 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36cbfff700 1 -- 172.21.15.88:0/2147862380 learned_addr learned my addr 172.21.15.88:0/2147862380 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36caffd700 1 -- 172.21.15.88:0/2147862380 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3097380007 0 0) 0x7f36cc108f10 con 0x7f36cc104ae0 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36caffd700 1 -- 172.21.15.88:0/2147862380 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36b40035b0 con 0x7f36cc104ae0 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36caffd700 1 -- 172.21.15.88:0/2147862380 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3845306742 0 0) 0x7f36b40035b0 con 0x7f36cc104ae0 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36caffd700 1 -- 172.21.15.88:0/2147862380 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36cc10a0f0 con 0x7f36cc104ae0 2024-03-26T21:15:49.073 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36caffd700 1 -- 172.21.15.88:0/2147862380 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f36bc0018c0 con 0x7f36cc104ae0 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36caffd700 1 -- 172.21.15.88:0/2147862380 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f36bc001bf0 con 0x7f36cc104ae0 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.437+0000 7f36caffd700 1 -- 172.21.15.88:0/2147862380 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f36bc0037b0 con 0x7f36cc104ae0 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.438+0000 7f36d26f2700 1 -- 172.21.15.88:0/2147862380 >> v1:172.21.15.88:6789/0 conn(0x7f36cc104ae0 legacy=0x7f36cc104f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.438+0000 7f36d26f2700 1 -- 172.21.15.88:0/2147862380 shutdown_connections 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.438+0000 7f36d26f2700 1 -- 172.21.15.88:0/2147862380 >> 172.21.15.88:0/2147862380 conn(0x7f36cc100040 msgr2=0x7f36cc1024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.438+0000 7f36d26f2700 1 -- 172.21.15.88:0/2147862380 shutdown_connections 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.438+0000 7f36d26f2700 1 -- 172.21.15.88:0/2147862380 wait complete. 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.438+0000 7f36d26f2700 1 Processor -- start 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.438+0000 7f36d26f2700 1 -- start start 2024-03-26T21:15:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36d26f2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f36cc19f860 con 0x7f36cc104ae0 2024-03-26T21:15:49.075 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36cbfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f36cc104ae0 0x7f36cc101f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50530/0 (socket says 172.21.15.88:50530) 2024-03-26T21:15:49.075 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36cbfff700 1 -- 172.21.15.88:0/2489535387 learned_addr learned my addr 172.21.15.88:0/2489535387 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:49.075 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722381912 0 0) 0x7f36cc19f860 con 0x7f36cc104ae0 2024-03-26T21:15:49.075 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36b40031f0 con 0x7f36cc104ae0 2024-03-26T21:15:49.075 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1875170346 0 0) 0x7f36b40031f0 con 0x7f36cc104ae0 2024-03-26T21:15:49.075 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f36cc19f860 con 0x7f36cc104ae0 2024-03-26T21:15:49.075 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f36bc0017c0 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2581309502 0 0) 0x7f36cc19f860 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36cc1a0a40 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.439+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f36cc19fa90 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.440+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f36cc19ffd0 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.440+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f36bc003de0 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.440+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f36bc0040a0 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.440+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f36bc005830 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.440+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7f36bc006e30 con 0x7f36cc104ae0 2024-03-26T21:15:49.076 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.440+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36cc063c70 con 0x7f36cc104ae0 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.443+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f36bc005d00 con 0x7f36cc104ae0 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.510+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f36cc109220 con 0x7f36cc104ae0 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.512+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+317 (unknown 1213389831 0 895421454) 0x7f36bc005f60 con 0x7f36cc104ae0 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.513+0000 7f36c97fa700 1 -- 172.21.15.88:0/2489535387 <== mon.0 v1:172.21.15.88:6789/0 11 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f36bc0065b0 con 0x7f36cc104ae0 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.513+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 >> v1:172.21.15.88:6789/0 conn(0x7f36cc104ae0 legacy=0x7f36cc101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.513+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 shutdown_connections 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.513+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 >> 172.21.15.88:0/2489535387 conn(0x7f36cc100040 msgr2=0x7f36cc107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.513+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 shutdown_connections 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:48.513+0000 7f36d26f2700 1 -- 172.21.15.88:0/2489535387 wait complete. 2024-03-26T21:15:49.077 INFO:teuthology.orchestra.run.smithi088.stdout:Generating new minimal ceph.conf... 2024-03-26T21:15:50.131 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:49 smithi088 ceph-mon[29599]: from='client.? v1:172.21.15.88:0/2402704613' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:15:50.137 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.482+0000 7f4c658c2700 1 Processor -- start 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.482+0000 7f4c658c2700 1 -- start start 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.482+0000 7f4c658c2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4c6007a6f0 con 0x7f4c60106f30 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.482+0000 7f4c5effd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4c60106f30 0x7f4c60109320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50536/0 (socket says 172.21.15.88:50536) 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.482+0000 7f4c5effd700 1 -- 172.21.15.88:0/1572866843 learned_addr learned my addr 172.21.15.88:0/1572866843 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c5e7fc700 1 -- 172.21.15.88:0/1572866843 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695919978 0 0) 0x7f4c6007a6f0 con 0x7f4c60106f30 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c5e7fc700 1 -- 172.21.15.88:0/1572866843 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c0035b0 con 0x7f4c60106f30 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c5e7fc700 1 -- 172.21.15.88:0/1572866843 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2561018278 0 0) 0x7f4c4c0035b0 con 0x7f4c60106f30 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c5e7fc700 1 -- 172.21.15.88:0/1572866843 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6010ba50 con 0x7f4c60106f30 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c5e7fc700 1 -- 172.21.15.88:0/1572866843 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f4c50001580 con 0x7f4c60106f30 2024-03-26T21:15:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c5e7fc700 1 -- 172.21.15.88:0/1572866843 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f4c50001d20 con 0x7f4c60106f30 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c5e7fc700 1 -- 172.21.15.88:0/1572866843 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f4c50005960 con 0x7f4c60106f30 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.483+0000 7f4c658c2700 1 -- 172.21.15.88:0/1572866843 >> v1:172.21.15.88:6789/0 conn(0x7f4c60106f30 legacy=0x7f4c60109320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.484+0000 7f4c658c2700 1 -- 172.21.15.88:0/1572866843 shutdown_connections 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.484+0000 7f4c658c2700 1 -- 172.21.15.88:0/1572866843 >> 172.21.15.88:0/1572866843 conn(0x7f4c601008e0 msgr2=0x7f4c60102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.484+0000 7f4c658c2700 1 -- 172.21.15.88:0/1572866843 shutdown_connections 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.484+0000 7f4c658c2700 1 -- 172.21.15.88:0/1572866843 wait complete. 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.484+0000 7f4c658c2700 1 Processor -- start 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.484+0000 7f4c658c2700 1 -- start start 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c658c2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4c6019a7c0 con 0x7f4c60106f30 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c5effd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4c60106f30 0x7f4c6019a0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50538/0 (socket says 172.21.15.88:50538) 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c5effd700 1 -- 172.21.15.88:0/2402704613 learned_addr learned my addr 172.21.15.88:0/2402704613 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1201061169 0 0) 0x7f4c6019a7c0 con 0x7f4c60106f30 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c005530 con 0x7f4c60106f30 2024-03-26T21:15:50.139 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3491910052 0 0) 0x7f4c4c005530 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c6019a7c0 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f4c50001080 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1527255030 0 0) 0x7f4c6019a7c0 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6019a990 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6019ac80 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c6019e7d0 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f4c50004ab0 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f4c50005f30 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4c50006580 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.485+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7f4c50007ce0 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.486+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c40005120 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.488+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4c500056d0 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.555+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f4c40005410 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.555+0000 7f4c648c0700 1 -- 172.21.15.88:0/2402704613 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+147 (unknown 1452402520 0 2370960669) 0x7f4c50005950 con 0x7f4c60106f30 2024-03-26T21:15:50.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.556+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 >> v1:172.21.15.88:6789/0 conn(0x7f4c60106f30 legacy=0x7f4c6019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:50.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.556+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 shutdown_connections 2024-03-26T21:15:50.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.556+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 >> 172.21.15.88:0/2402704613 conn(0x7f4c601008e0 msgr2=0x7f4c60190d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:50.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.556+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 shutdown_connections 2024-03-26T21:15:50.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:49.556+0000 7f4c658c2700 1 -- 172.21.15.88:0/2402704613 wait complete. 2024-03-26T21:15:50.141 INFO:teuthology.orchestra.run.smithi088.stdout:Restarting the monitor... 2024-03-26T21:15:50.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:50 smithi088 systemd[1]: Stopping Ceph mon.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:15:50.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a[29576]: 2024-03-26T21:15:50.410+0000 7f34b6373700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:15:50.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a[29576]: 2024-03-26T21:15:50.410+0000 7f34b6373700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T21:15:50.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:50 smithi088 podman[29921]: 2024-03-26 21:15:50.523298237 +0000 UTC m=+0.224965132 container died aadec77452d443da5a811432ed657797aa6948b8556cb5eb7e681397d820e8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:15:51.043 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:50 smithi088 podman[29921]: 2024-03-26 21:15:50.849435728 +0000 UTC m=+0.551102693 container cleanup aadec77452d443da5a811432ed657797aa6948b8556cb5eb7e681397d820e8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T21:15:51.043 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:50 smithi088 bash[29921]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a 2024-03-26T21:15:51.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 podman[29940]: 2024-03-26 21:15:51.050259091 +0000 UTC m=+0.527076861 container remove aadec77452d443da5a811432ed657797aa6948b8556cb5eb7e681397d820e8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T21:15:51.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T21:15:51.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 systemd[1]: Stopped Ceph mon.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:15:51.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 systemd[1]: Starting Ceph mon.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:15:51.675 INFO:teuthology.orchestra.run.smithi088.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 podman[30038]: 2024-03-26 21:15:51.451150288 +0000 UTC m=+0.094516717 container create e7a95e70ceca0ad07fa04ad6629e159bc0d7bae26810f5b3d677232cbe8acdce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 podman[30038]: 2024-03-26 21:15:51.373928322 +0000 UTC m=+0.017294709 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 podman[30038]: 2024-03-26 21:15:51.611980093 +0000 UTC m=+0.255346489 container init e7a95e70ceca0ad07fa04ad6629e159bc0d7bae26810f5b3d677232cbe8acdce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 podman[30038]: 2024-03-26 21:15:51.617354271 +0000 UTC m=+0.260720668 container start e7a95e70ceca0ad07fa04ad6629e159bc0d7bae26810f5b3d677232cbe8acdce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev), process ceph-mon, pid 7 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: pidfile_write: ignore empty --pid-file 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: load: jerasure load: lrc 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: RocksDB version: 7.9.2 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Git sha 0 2024-03-26T21:15:51.711 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Compile date 2024-03-26 11:42:14 2024-03-26T21:15:51.712 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: DB SUMMARY 2024-03-26T21:15:51.712 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: DB Session ID: QQSWGKGZUPY8Q9TQN660 2024-03-26T21:15:51.712 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: CURRENT file: CURRENT 2024-03-26T21:15:51.712 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T21:15:51.712 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T21:15:51.712 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T21:15:51.712 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 89185 ; 2024-03-26T21:15:51.713 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.error_if_exists: 0 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.create_if_missing: 0 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.paranoid_checks: 1 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.env: 0x55a88dc42c60 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.info_log: 0x55a88fb5db20 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.statistics: (nil) 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.use_fsync: 0 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_log_file_size: 0 2024-03-26T21:15:51.714 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.allow_fallocate: 1 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.use_direct_reads: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.db_log_dir: 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.wal_dir: 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T21:15:51.715 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.write_buffer_manager: 0x55a88fb2e1e0 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.unordered_write: 0 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T21:15:51.716 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.row_cache: None 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.wal_filter: None 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.two_write_queues: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.wal_compression: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.atomic_flush: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.log_readahead_size: 0 2024-03-26T21:15:51.717 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_background_jobs: 2 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_background_compactions: -1 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_subcompactions: 1 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T21:15:51.718 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_open_files: -1 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_background_flushes: -1 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Compression algorithms supported: 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kZSTD supported: 0 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kXpressCompression supported: 0 2024-03-26T21:15:51.719 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kZlibCompression supported: 1 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kSnappyCompression supported: 1 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kLZ4Compression supported: 1 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: kBZip2Compression supported: 0 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.merge_operator: 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_filter: None 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T21:15:51.720 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a88fb5dd00) 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: cache_index_and_filter_blocks: 1 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: pin_top_level_index_and_filter: 1 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: index_type: 0 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: data_block_index_type: 0 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: index_shortening: 1 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: checksum: 4 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: no_block_cache: 0 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache: 0x55a88fb61090 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache_name: BinnedLRUCache 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache_options: 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: capacity : 536870912 2024-03-26T21:15:51.721 INFO:journalctl@ceph.mon.a.smithi088.stdout: num_shard_bits : 4 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: strict_capacity_limit : 0 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: high_pri_pool_ratio: 0.000 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache_compressed: (nil) 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: persistent_cache: (nil) 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_size: 4096 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_size_deviation: 10 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_restart_interval: 16 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: index_block_restart_interval: 1 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: metadata_block_size: 4096 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: partition_filters: 0 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: use_delta_encoding: 1 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: filter_policy: bloomfilter 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: whole_key_filtering: 1 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: verify_compression: 0 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: read_amp_bytes_per_bit: 0 2024-03-26T21:15:51.722 INFO:journalctl@ceph.mon.a.smithi088.stdout: format_version: 5 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout: enable_index_compression: 1 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_align: 0 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout: max_auto_readahead_size: 262144 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout: prepopulate_block_cache: 0 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout: initial_auto_readahead_size: 8192 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression: NoCompression 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.num_levels: 7 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T21:15:51.723 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:15:51.724 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T21:15:51.725 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T21:15:51.726 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.inplace_update_support: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.bloom_locality: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.max_successive_merges: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T21:15:51.727 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.ttl: 2592000 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.enable_blob_files: false 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.min_blob_size: 0 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T21:15:51.728 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c4306ee9-b528-496b-8603-c4c7262b055e 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487751646298, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487751647168, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85494, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 270, "table_properties": {"data_size": 83647, "index_size": 231, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10877, "raw_average_key_size": 47, "raw_value_size": 77546, "raw_average_value_size": 338, "num_data_blocks": 10, "num_entries": 229, "num_filter_entries": 229, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711487751, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c4306ee9-b528-496b-8603-c4c7262b055e", "db_session_id": "QQSWGKGZUPY8Q9TQN660", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487751647255, "job": 1, "event": "recovery_finished"} 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a88fc50000 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: DB pointer 0x55a88fc3a000 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** DB Stats ** 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:15:51.729 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** Compaction Stats [default] ** 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: L0 2/0 85.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Sum 2/0 85.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** Compaction Stats [default] ** 2024-03-26T21:15:51.730 INFO:journalctl@ceph.mon.a.smithi088.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T21:15:51.731 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative compaction: 0.00 GB write, 28.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval compaction: 0.00 GB write, 28.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: Block cache BinnedLRUCache@0x55a88fb61090#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: starting mon.a rank 0 at public addrs v1:172.21.15.88:6789/0 at bind addrs v1:172.21.15.88:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???) e1 preinit fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???).mds e1 new map 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???).mds e1 print_map 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: e1 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T21:15:51.732 INFO:journalctl@ceph.mon.a.smithi088.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout: legacy client fscid: -1 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout: No filesystems configured 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 bash[30038]: e7a95e70ceca0ad07fa04ad6629e159bc0d7bae26810f5b3d677232cbe8acdce 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 systemd[1]: Started Ceph mon.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: monmap epoch 1 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: last_changed 2024-03-26T21:15:43.819826+0000 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:15:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: min_mon_release 19 (squid) 2024-03-26T21:15:51.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: election_strategy: 1 2024-03-26T21:15:51.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:15:51.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: fsmap 2024-03-26T21:15:51.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T21:15:51.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:51 smithi088 ceph-mon[30075]: mgrmap e1: no daemons active 2024-03-26T21:15:52.776 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.183+0000 7fb63bbf1700 1 Processor -- start 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.184+0000 7fb63bbf1700 1 -- start start 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.184+0000 7fb63bbf1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb63407b360 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.184+0000 7fb63998d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb63407c760 0x7fb63407abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50550/0 (socket says 172.21.15.88:50550) 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.184+0000 7fb63998d700 1 -- 172.21.15.88:0/3107546839 learned_addr learned my addr 172.21.15.88:0/3107546839 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.184+0000 7fb63898b700 1 -- 172.21.15.88:0/3107546839 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134127210 0 0) 0x7fb63407b360 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.184+0000 7fb63898b700 1 -- 172.21.15.88:0/3107546839 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6200035b0 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63898b700 1 -- 172.21.15.88:0/3107546839 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3013760545 0 0) 0x7fb6200035b0 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63898b700 1 -- 172.21.15.88:0/3107546839 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63407b530 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63898b700 1 -- 172.21.15.88:0/3107546839 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb624001580 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63898b700 1 -- 172.21.15.88:0/3107546839 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fb624001d20 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63898b700 1 -- 172.21.15.88:0/3107546839 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb624005960 con 0x7fb63407c760 2024-03-26T21:15:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63bbf1700 1 -- 172.21.15.88:0/3107546839 >> v1:172.21.15.88:6789/0 conn(0x7fb63407c760 legacy=0x7fb63407abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63bbf1700 1 -- 172.21.15.88:0/3107546839 shutdown_connections 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.185+0000 7fb63bbf1700 1 -- 172.21.15.88:0/3107546839 >> 172.21.15.88:0/3107546839 conn(0x7fb634101960 msgr2=0x7fb634103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb63bbf1700 1 -- 172.21.15.88:0/3107546839 shutdown_connections 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb63bbf1700 1 -- 172.21.15.88:0/3107546839 wait complete. 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb63bbf1700 1 Processor -- start 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb63bbf1700 1 -- start start 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb63bbf1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb63419c9c0 con 0x7fb63407c760 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb63998d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb63407c760 0x7fb63419c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50558/0 (socket says 172.21.15.88:50558) 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb63998d700 1 -- 172.21.15.88:0/16195043 learned_addr learned my addr 172.21.15.88:0/16195043 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3448576845 0 0) 0x7fb63419c9c0 con 0x7fb63407c760 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.186+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb620005530 con 0x7fb63407c760 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2590355028 0 0) 0x7fb620005530 con 0x7fb63407c760 2024-03-26T21:15:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb63419c9c0 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb624001080 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4202884807 0 0) 0x7fb63419c9c0 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63419cb90 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb63419ce80 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6341a09d0 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fb624001c60 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb624005fa0 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb624006590 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.187+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7fb624007ca0 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.188+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb618005150 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.190+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb624004ab0 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.257+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 --> v1:172.21.15.88:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fb618005440 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.259+0000 7fb62affd700 1 -- 172.21.15.88:0/16195043 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (unknown 534127980 0 0) 0x7fb624004d10 con 0x7fb63407c760 2024-03-26T21:15:52.779 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.260+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 >> v1:172.21.15.88:6789/0 conn(0x7fb63407c760 legacy=0x7fb63419c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:52.780 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.260+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 shutdown_connections 2024-03-26T21:15:52.780 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.260+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 >> 172.21.15.88:0/16195043 conn(0x7fb634101960 msgr2=0x7fb63410a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:52.780 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.260+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 shutdown_connections 2024-03-26T21:15:52.780 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:52.260+0000 7fb63bbf1700 1 -- 172.21.15.88:0/16195043 wait complete. 2024-03-26T21:15:52.780 INFO:teuthology.orchestra.run.smithi088.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T21:15:52.795 INFO:teuthology.orchestra.run.smithi088.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:15:52.795 INFO:teuthology.orchestra.run.smithi088.stdout:Creating mgr... 2024-03-26T21:15:52.795 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T21:15:52.796 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T21:15:53.037 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.y 2024-03-26T21:15:53.038 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T21:15:53.186 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715.target.wants/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@.service. 2024-03-26T21:15:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/16195043' entity='client.admin' 2024-03-26T21:15:53.485 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:53 smithi088 podman[30319]: 2024-03-26 21:15:53.437651947 +0000 UTC m=+0.124878157 container create c1848489286877ee9f2a68c12da1cd5071d063257c79dc9bbf5a6f186fa526ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T21:15:53.764 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:15:53.764 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2024-03-26T21:15:53.771 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:15:53.771 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2024-03-26T21:15:53.771 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2024-03-26T21:15:53.771 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T21:15:53.779 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:15:53.779 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2024-03-26T21:15:53.787 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:15:53.787 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2024-03-26T21:15:53.787 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2024-03-26T21:15:53.787 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T21:15:53.787 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr to start... 2024-03-26T21:15:53.787 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr... 2024-03-26T21:15:53.934 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:53 smithi088 podman[30319]: 2024-03-26 21:15:53.690091588 +0000 UTC m=+0.377317753 container init c1848489286877ee9f2a68c12da1cd5071d063257c79dc9bbf5a6f186fa526ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T21:15:53.934 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:53 smithi088 podman[30319]: 2024-03-26 21:15:53.695481392 +0000 UTC m=+0.382707557 container start c1848489286877ee9f2a68c12da1cd5071d063257c79dc9bbf5a6f186fa526ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T21:15:53.934 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:53 smithi088 bash[30319]: c1848489286877ee9f2a68c12da1cd5071d063257c79dc9bbf5a6f186fa526ef 2024-03-26T21:15:53.934 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:53 smithi088 systemd[1]: Started Ceph mgr.y for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:15:54.639 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:15:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/277173078' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:15:54.639 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:54.246+0000 7f3586043200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:15:54.639 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:54.342+0000 7f3586043200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsid": "e240a1a4-ebb5-11ee-95d0-87774f69a715", 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:15:54.921 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:15:54.922 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:15:54.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:15:46.786562+0000", 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:15:54.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.253+0000 7fb81c10c700 1 Processor -- start 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.253+0000 7fb81c10c700 1 -- start start 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.253+0000 7fb81c10c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb814079e60 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.253+0000 7fb819ea8700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb814106f30 0x7fb814109320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36602/0 (socket says 172.21.15.88:36602) 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.253+0000 7fb819ea8700 1 -- 172.21.15.88:0/3814660188 learned_addr learned my addr 172.21.15.88:0/3814660188 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.254+0000 7fb818ea6700 1 -- 172.21.15.88:0/3814660188 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719348220 0 0) 0x7fb814079e60 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.254+0000 7fb818ea6700 1 -- 172.21.15.88:0/3814660188 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8000035b0 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.254+0000 7fb818ea6700 1 -- 172.21.15.88:0/3814660188 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1616146093 0 0) 0x7fb8000035b0 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.254+0000 7fb818ea6700 1 -- 172.21.15.88:0/3814660188 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81410ba50 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.254+0000 7fb818ea6700 1 -- 172.21.15.88:0/3814660188 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb8100018c0 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.254+0000 7fb818ea6700 1 -- 172.21.15.88:0/3814660188 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fb810001b20 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.254+0000 7fb818ea6700 1 -- 172.21.15.88:0/3814660188 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb810005a60 con 0x7fb814106f30 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.255+0000 7fb81c10c700 1 -- 172.21.15.88:0/3814660188 >> v1:172.21.15.88:6789/0 conn(0x7fb814106f30 legacy=0x7fb814109320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:54.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.255+0000 7fb81c10c700 1 -- 172.21.15.88:0/3814660188 shutdown_connections 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.255+0000 7fb81c10c700 1 -- 172.21.15.88:0/3814660188 >> 172.21.15.88:0/3814660188 conn(0x7fb8141008c0 msgr2=0x7fb814102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.255+0000 7fb81c10c700 1 -- 172.21.15.88:0/3814660188 shutdown_connections 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.255+0000 7fb81c10c700 1 -- 172.21.15.88:0/3814660188 wait complete. 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb81c10c700 1 Processor -- start 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb81c10c700 1 -- start start 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb81c10c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb81419a7d0 con 0x7fb814106f30 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb819ea8700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb814106f30 0x7fb81419a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36606/0 (socket says 172.21.15.88:36606) 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb819ea8700 1 -- 172.21.15.88:0/277173078 learned_addr learned my addr 172.21.15.88:0/277173078 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2070107648 0 0) 0x7fb81419a7d0 con 0x7fb814106f30 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb800005580 con 0x7fb814106f30 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1316523598 0 0) 0x7fb800005580 con 0x7fb814106f30 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb81419a7d0 con 0x7fb814106f30 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb8100013c0 con 0x7fb814106f30 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2511503264 0 0) 0x7fb81419a7d0 con 0x7fb814106f30 2024-03-26T21:15:54.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81419a9a0 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.256+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb81419ac90 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.257+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb81419e7e0 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.257+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fb810001960 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.257+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fb8100060f0 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.257+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb810006700 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.257+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7fb810007dc0 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.257+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f8005120 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.260+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb810003680 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.332+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb7f8005410 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.332+0000 7fb80affd700 1 -- 172.21.15.88:0/277173078 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 47571803) 0x7fb810003940 con 0x7fb814106f30 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.333+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 >> v1:172.21.15.88:6789/0 conn(0x7fb814106f30 legacy=0x7fb81419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.333+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 shutdown_connections 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.333+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 >> 172.21.15.88:0/277173078 conn(0x7fb8141008c0 msgr2=0x7fb814190d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.333+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 shutdown_connections 2024-03-26T21:15:54.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:54.333+0000 7fb81c10c700 1 -- 172.21.15.88:0/277173078 wait complete. 2024-03-26T21:15:54.928 INFO:teuthology.orchestra.run.smithi088.stdout:mgr not available, waiting (1/15)... 2024-03-26T21:15:55.211 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:54.991+0000 7f3586043200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:15:55.212 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.080+0000 7f3586043200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:15:55.212 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.217+0000 7f3586043200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:15:55.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.310+0000 7f3586043200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:15:55.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.389+0000 7f3586043200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:15:55.922 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.647+0000 7f3586043200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:15:55.923 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.752+0000 7f3586043200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:15:55.923 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.836+0000 7f3586043200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:15:55.923 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:55 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:55.928+0000 7f3586043200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:15:56.234 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:56 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:56.101+0000 7f3586043200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:15:57.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:57 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:57.059+0000 7f3586043200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:15:57.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:57 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:57.252+0000 7f3586043200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:15:57.851 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:57 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:57.578+0000 7f3586043200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:15:57.851 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:57 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:57.664+0000 7f3586043200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:15:57.851 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:57 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:57.856+0000 7f3586043200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:15:58.234 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:58 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:58.036+0000 7f3586043200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:15:58.588 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:58 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:58.314+0000 7f3586043200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:15:58.588 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:58 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:58.593+0000 7f3586043200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:15:58.922 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:58 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:58.826+0000 7f3586043200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:15:59.890 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:15:59 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:15:59.628+0000 7f3586043200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:16:00.202 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/257102200' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:16:00.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:00 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:00.208+0000 7f3586043200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:16:00.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:00 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:00.305+0000 7f3586043200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:16:00.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:00 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:00.521+0000 7f3586043200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:16:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: Activating manager daemon y 2024-03-26T21:16:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: mgrmap e2: y(active, starting, since 0.00297991s) 2024-03-26T21:16:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:16:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: Manager daemon y is now available 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' 2024-03-26T21:16:01.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:01 smithi088 ceph-mon[30075]: from='mgr.14100 v1:172.21.15.88:0/1822898183' entity='mgr.y' 2024-03-26T21:16:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:02 smithi088 ceph-mon[30075]: mgrmap e3: y(active, since 1.00694s) 2024-03-26T21:16:04.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:04 smithi088 ceph-mon[30075]: mgrmap e4: y(active, since 3s) 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsid": "e240a1a4-ebb5-11ee-95d0-87774f69a715", 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:16:05.951 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:16:05.952 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:16:05.953 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:16:05.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:16:05.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:16:05.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:16:05.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:15:46.786562+0000", 2024-03-26T21:16:05.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.996+0000 7f770559f700 1 Processor -- start 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.996+0000 7f770559f700 1 -- start start 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.996+0000 7f770559f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f770007a710 con 0x7f7700106d40 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.996+0000 7f76fffff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7700106d40 0x7f7700109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36610/0 (socket says 172.21.15.88:36610) 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.996+0000 7f76fffff700 1 -- 172.21.15.88:0/189520797 learned_addr learned my addr 172.21.15.88:0/189520797 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.997+0000 7f76feffd700 1 -- 172.21.15.88:0/189520797 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338466699 0 0) 0x7f770007a710 con 0x7f7700106d40 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.997+0000 7f76feffd700 1 -- 172.21.15.88:0/189520797 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76e80035b0 con 0x7f7700106d40 2024-03-26T21:16:05.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.997+0000 7f76feffd700 1 -- 172.21.15.88:0/189520797 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3714710326 0 0) 0x7f76e80035b0 con 0x7f7700106d40 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.997+0000 7f76feffd700 1 -- 172.21.15.88:0/189520797 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770010b860 con 0x7f7700106d40 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.997+0000 7f76feffd700 1 -- 172.21.15.88:0/189520797 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f76f4001580 con 0x7f7700106d40 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.997+0000 7f76feffd700 1 -- 172.21.15.88:0/189520797 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f76f4001c50 con 0x7f7700106d40 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.997+0000 7f76feffd700 1 -- 172.21.15.88:0/189520797 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f76f4005960 con 0x7f7700106d40 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.998+0000 7f770559f700 1 -- 172.21.15.88:0/189520797 >> v1:172.21.15.88:6789/0 conn(0x7f7700106d40 legacy=0x7f7700109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.998+0000 7f770559f700 1 -- 172.21.15.88:0/189520797 shutdown_connections 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.998+0000 7f770559f700 1 -- 172.21.15.88:0/189520797 >> 172.21.15.88:0/189520797 conn(0x7f7700100750 msgr2=0x7f7700102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.998+0000 7f770559f700 1 -- 172.21.15.88:0/189520797 shutdown_connections 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.998+0000 7f770559f700 1 -- 172.21.15.88:0/189520797 wait complete. 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.998+0000 7f770559f700 1 Processor -- start 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.998+0000 7f770559f700 1 -- start start 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f770559f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f770019a5d0 con 0x7f7700106d40 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fffff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7700106d40 0x7f7700199ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36624/0 (socket says 172.21.15.88:36624) 2024-03-26T21:16:05.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fffff700 1 -- 172.21.15.88:0/257102200 learned_addr learned my addr 172.21.15.88:0/257102200 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095592620 0 0) 0x7f770019a5d0 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76e8005580 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 217130784 0 0) 0x7f76e8005580 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f770019a5d0 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f76f4001080 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:15:59.999+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1567527432 0 0) 0x7f770019a5d0 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.000+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770019a7a0 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.000+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f770019aa90 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.000+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f770019e5e0 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.000+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f76f4001c10 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.000+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f76f4006030 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.000+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f76f4006640 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.000+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7f76f4007d50 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.001+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76ec005120 con 0x7f7700106d40 2024-03-26T21:16:05.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.003+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f76f40055f0 con 0x7f7700106d40 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.075+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f76ec005410 con 0x7f7700106d40 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.075+0000 7f76fd7fa700 1 -- 172.21.15.88:0/257102200 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 3818631716) 0x7f76f4005850 con 0x7f7700106d40 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.077+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 >> v1:172.21.15.88:6789/0 conn(0x7f7700106d40 legacy=0x7f7700199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.077+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 shutdown_connections 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.077+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 >> 172.21.15.88:0/257102200 conn(0x7f7700100750 msgr2=0x7f7700190bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.077+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 shutdown_connections 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:00.077+0000 7f770559f700 1 -- 172.21.15.88:0/257102200 wait complete. 2024-03-26T21:16:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:mgr not available, waiting (2/15)... 2024-03-26T21:16:08.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2764235393' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:16:09.240 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsid": "e240a1a4-ebb5-11ee-95d0-87774f69a715", 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:16:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:16:09.242 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:16:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:16:09.251 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:15:46.786562+0000", 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.451+0000 7f727ed94700 1 Processor -- start 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.451+0000 7f727ed94700 1 -- start start 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.451+0000 7f727ed94700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f727810a810 con 0x7f72781063e0 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.451+0000 7f727cb30700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f72781063e0 0x7f7278106800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60648/0 (socket says 172.21.15.88:60648) 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.451+0000 7f727cb30700 1 -- 172.21.15.88:0/1157154637 learned_addr learned my addr 172.21.15.88:0/1157154637 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:09.252 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f72777fe700 1 -- 172.21.15.88:0/1157154637 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1563157982 0 0) 0x7f727810a810 con 0x7f72781063e0 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f72777fe700 1 -- 172.21.15.88:0/1157154637 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72680035b0 con 0x7f72781063e0 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f72777fe700 1 -- 172.21.15.88:0/1157154637 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1693787635 0 0) 0x7f72680035b0 con 0x7f72781063e0 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f72777fe700 1 -- 172.21.15.88:0/1157154637 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727810b9f0 con 0x7f72781063e0 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f72777fe700 1 -- 172.21.15.88:0/1157154637 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f726c001580 con 0x7f72781063e0 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f72777fe700 1 -- 172.21.15.88:0/1157154637 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f726c001d20 con 0x7f72781063e0 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f72777fe700 1 -- 172.21.15.88:0/1157154637 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f726c005960 con 0x7f72781063e0 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f727ed94700 1 -- 172.21.15.88:0/1157154637 >> v1:172.21.15.88:6789/0 conn(0x7f72781063e0 legacy=0x7f7278106800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f727ed94700 1 -- 172.21.15.88:0/1157154637 shutdown_connections 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f727ed94700 1 -- 172.21.15.88:0/1157154637 >> 172.21.15.88:0/1157154637 conn(0x7f7278101940 msgr2=0x7f7278103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f727ed94700 1 -- 172.21.15.88:0/1157154637 shutdown_connections 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.452+0000 7f727ed94700 1 -- 172.21.15.88:0/1157154637 wait complete. 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.453+0000 7f727ed94700 1 Processor -- start 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.453+0000 7f727ed94700 1 -- start start 2024-03-26T21:16:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.453+0000 7f727ed94700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f727819a6e0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.453+0000 7f727cb30700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f72781063e0 0x7f7278199fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60654/0 (socket says 172.21.15.88:60654) 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.453+0000 7f727cb30700 1 -- 172.21.15.88:0/2764235393 learned_addr learned my addr 172.21.15.88:0/2764235393 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.453+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205550311 0 0) 0x7f727819a6e0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.453+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7268005530 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 552348402 0 0) 0x7f7268005530 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f727819a6e0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f726c004870 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1300670104 0 0) 0x7f727819a6e0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727819a8b0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f727819aba0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f726c004a40 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f726c0061c0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.454+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f727819e6f0 con 0x7f72781063e0 2024-03-26T21:16:09.254 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.455+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) ==== 46420+0+0 (unknown 3758884248 0 0) 0x7f726c011a30 con 0x7f72781063e0 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.455+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7f726c047c70 con 0x7f72781063e0 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.455+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7278063c70 con 0x7f72781063e0 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.460+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f726c017550 con 0x7f72781063e0 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.736+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7278193470 con 0x7f72781063e0 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.737+0000 7f7275ffb700 1 -- 172.21.15.88:0/2764235393 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 1805297507) 0x7f7278193470 con 0x7f72781063e0 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.740+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 >> v1:172.21.15.88:6800/221932617 conn(0x7f726803a650 legacy=0x7f726803cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.740+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 >> v1:172.21.15.88:6789/0 conn(0x7f72781063e0 legacy=0x7f7278199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.740+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 shutdown_connections 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.740+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 >> 172.21.15.88:0/2764235393 conn(0x7f7278101940 msgr2=0x7f7278103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.740+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 shutdown_connections 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:08.740+0000 7f727ed94700 1 -- 172.21.15.88:0/2764235393 wait complete. 2024-03-26T21:16:09.255 INFO:teuthology.orchestra.run.smithi088.stdout:mgr is available 2024-03-26T21:16:11.317 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/588882247' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T21:16:11.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout fsid = e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T21:16:11.609 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.789+0000 7ff65f6a8700 1 Processor -- start 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.790+0000 7ff65f6a8700 1 -- start start 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.790+0000 7ff65f6a8700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6581097a0 con 0x7ff658105370 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.790+0000 7ff65d444700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff658105370 0x7ff658105790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60656/0 (socket says 172.21.15.88:60656) 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.790+0000 7ff65d444700 1 -- 172.21.15.88:0/4252029721 learned_addr learned my addr 172.21.15.88:0/4252029721 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.790+0000 7ff64ffff700 1 -- 172.21.15.88:0/4252029721 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1694426025 0 0) 0x7ff6581097a0 con 0x7ff658105370 2024-03-26T21:16:11.610 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.790+0000 7ff64ffff700 1 -- 172.21.15.88:0/4252029721 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6440035b0 con 0x7ff658105370 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff64ffff700 1 -- 172.21.15.88:0/4252029721 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3422482524 0 0) 0x7ff6440035b0 con 0x7ff658105370 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff64ffff700 1 -- 172.21.15.88:0/4252029721 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65810a980 con 0x7ff658105370 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff64ffff700 1 -- 172.21.15.88:0/4252029721 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ff654001580 con 0x7ff658105370 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff64ffff700 1 -- 172.21.15.88:0/4252029721 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7ff654001d20 con 0x7ff658105370 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff64ffff700 1 -- 172.21.15.88:0/4252029721 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ff654005960 con 0x7ff658105370 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff65f6a8700 1 -- 172.21.15.88:0/4252029721 >> v1:172.21.15.88:6789/0 conn(0x7ff658105370 legacy=0x7ff658105790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff65f6a8700 1 -- 172.21.15.88:0/4252029721 shutdown_connections 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff65f6a8700 1 -- 172.21.15.88:0/4252029721 >> 172.21.15.88:0/4252029721 conn(0x7ff658100930 msgr2=0x7ff658102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff65f6a8700 1 -- 172.21.15.88:0/4252029721 shutdown_connections 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.791+0000 7ff65f6a8700 1 -- 172.21.15.88:0/4252029721 wait complete. 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.792+0000 7ff65f6a8700 1 Processor -- start 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.792+0000 7ff65f6a8700 1 -- start start 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.792+0000 7ff65f6a8700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6581a0060 con 0x7ff658105370 2024-03-26T21:16:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.792+0000 7ff65d444700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff658105370 0x7ff6581027a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60668/0 (socket says 172.21.15.88:60668) 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.792+0000 7ff65d444700 1 -- 172.21.15.88:0/588882247 learned_addr learned my addr 172.21.15.88:0/588882247 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.792+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601721629 0 0) 0x7ff6581a0060 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.792+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff644005530 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 423008943 0 0) 0x7ff644005530 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6581a0060 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ff654004870 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2086218036 0 0) 0x7ff6581a0060 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6581a1240 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6581a0290 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6581a07d0 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7ff654001c00 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.793+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ff654005f10 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.794+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) ==== 46420+0+0 (unknown 3758884248 0 0) 0x7ff654011610 con 0x7ff658105370 2024-03-26T21:16:11.612 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.794+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63c005150 con 0x7ff658105370 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.794+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7ff6540489c0 con 0x7ff658105370 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:10.799+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7ff6540174d0 con 0x7ff658105370 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.006+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff63c002430 con 0x7ff658105370 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.007+0000 7ff64e7fc700 1 -- 172.21.15.88:0/588882247 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+282 (unknown 1187553405 0 108347180) 0x7ff654016dd0 con 0x7ff658105370 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.009+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 >> v1:172.21.15.88:6800/221932617 conn(0x7ff64403a650 legacy=0x7ff64403cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.009+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 >> v1:172.21.15.88:6789/0 conn(0x7ff658105370 legacy=0x7ff6581027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.009+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 shutdown_connections 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.009+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 >> 172.21.15.88:0/588882247 conn(0x7ff658100930 msgr2=0x7ff6580798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.009+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 shutdown_connections 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:11.009+0000 7ff65f6a8700 1 -- 172.21.15.88:0/588882247 wait complete. 2024-03-26T21:16:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling cephadm module... 2024-03-26T21:16:13.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2035938463' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T21:16:13.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:13 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ignoring --setuser ceph since I am not root 2024-03-26T21:16:13.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:13 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ignoring --setgroup ceph since I am not root 2024-03-26T21:16:13.866 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:13 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:13.779+0000 7f8a22548200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:16:13.866 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:13 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:13.872+0000 7f8a22548200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:16:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.907+0000 7fdbd57f9700 1 Processor -- start 2024-03-26T21:16:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.907+0000 7fdbd57f9700 1 -- start start 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.907+0000 7fdbd57f9700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbd0109790 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.907+0000 7fdbceffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdbd01068f0 0x7fdbd0106d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60672/0 (socket says 172.21.15.88:60672) 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.907+0000 7fdbceffd700 1 -- 172.21.15.88:0/3455994597 learned_addr learned my addr 172.21.15.88:0/3455994597 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.907+0000 7fdbcdffb700 1 -- 172.21.15.88:0/3455994597 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610436346 0 0) 0x7fdbd0109790 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.907+0000 7fdbcdffb700 1 -- 172.21.15.88:0/3455994597 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb80035b0 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbcdffb700 1 -- 172.21.15.88:0/3455994597 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3230079884 0 0) 0x7fdbb80035b0 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbcdffb700 1 -- 172.21.15.88:0/3455994597 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd010a970 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbcdffb700 1 -- 172.21.15.88:0/3455994597 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fdbc0001580 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbcdffb700 1 -- 172.21.15.88:0/3455994597 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fdbc0001d20 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbcdffb700 1 -- 172.21.15.88:0/3455994597 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fdbc0005960 con 0x7fdbd01068f0 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbd57f9700 1 -- 172.21.15.88:0/3455994597 >> v1:172.21.15.88:6789/0 conn(0x7fdbd01068f0 legacy=0x7fdbd0106d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbd57f9700 1 -- 172.21.15.88:0/3455994597 shutdown_connections 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbd57f9700 1 -- 172.21.15.88:0/3455994597 >> 172.21.15.88:0/3455994597 conn(0x7fdbd0100040 msgr2=0x7fdbd01024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbd57f9700 1 -- 172.21.15.88:0/3455994597 shutdown_connections 2024-03-26T21:16:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.908+0000 7fdbd57f9700 1 -- 172.21.15.88:0/3455994597 wait complete. 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.909+0000 7fdbd57f9700 1 Processor -- start 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.909+0000 7fdbd57f9700 1 -- start start 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.909+0000 7fdbd57f9700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbd01a00f0 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.909+0000 7fdbceffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdbd01068f0 0x7fdbd0101f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60684/0 (socket says 172.21.15.88:60684) 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.909+0000 7fdbceffd700 1 -- 172.21.15.88:0/2035938463 learned_addr learned my addr 172.21.15.88:0/2035938463 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153201075 0 0) 0x7fdbd01a00f0 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb8005530 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3242408214 0 0) 0x7fdbb8005530 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbd01a00f0 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fdbc0001080 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3731092159 0 0) 0x7fdbd01a00f0 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd01a12d0 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd01a0320 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbd01a0860 con 0x7fdbd01068f0 2024-03-26T21:16:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fdbc0004ab0 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.910+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fdbc0005f90 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.911+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) ==== 46420+0+0 (unknown 3758884248 0 0) 0x7fdbc0011690 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.911+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbbc005150 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.911+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7fdbc0047b70 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:12.916+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fdbc0017540 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.172+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fdbbc005440 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.223+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (unknown 2263024820 0 0) 0x7fdbc0016e40 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.225+0000 7fdbb7fff700 1 -- 172.21.15.88:0/2035938463 <== mon.0 v1:172.21.15.88:6789/0 11 ==== mgrmap(e 5) ==== 46431+0+0 (unknown 2967675786 0 0) 0x7fdbc0046560 con 0x7fdbd01068f0 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.226+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 >> v1:172.21.15.88:6800/221932617 conn(0x7fdbb803a7a0 legacy=0x7fdbb803cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.226+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 >> v1:172.21.15.88:6789/0 conn(0x7fdbd01068f0 legacy=0x7fdbd0101f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.226+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 shutdown_connections 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.226+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 >> 172.21.15.88:0/2035938463 conn(0x7fdbd0100040 msgr2=0x7fdbd0104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.226+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 shutdown_connections 2024-03-26T21:16:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:13.226+0000 7fdbd57f9700 1 -- 172.21.15.88:0/2035938463 wait complete. 2024-03-26T21:16:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2035938463' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T21:16:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:14 smithi088 ceph-mon[30075]: mgrmap e5: y(active, since 12s) 2024-03-26T21:16:14.863 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:14 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:14.536+0000 7f8a22548200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:16:14.864 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:14 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:14.630+0000 7f8a22548200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:16:14.864 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:14 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:14.775+0000 7f8a22548200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:16:14.864 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:14 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:14.869+0000 7f8a22548200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:16:15.211 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:14 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:14.956+0000 7f8a22548200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:16:15.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4169767790' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T21:16:15.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:15 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:15.217+0000 7f8a22548200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:16:15.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:15 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:15.327+0000 7f8a22548200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:16:15.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:15 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:15.412+0000 7f8a22548200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:16:15.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:16:15.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T21:16:15.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:16:15.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T21:16:15.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T21:16:15.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:16:15.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.361+0000 7fcfabcbc700 1 Processor -- start 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.361+0000 7fcfabcbc700 1 -- start start 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.361+0000 7fcfabcbc700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcfa4109790 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.361+0000 7fcfa9a58700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcfa4105360 0x7fcfa4105780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42610/0 (socket says 172.21.15.88:42610) 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.361+0000 7fcfa9a58700 1 -- 172.21.15.88:0/215771219 learned_addr learned my addr 172.21.15.88:0/215771219 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.361+0000 7fcfa8a56700 1 -- 172.21.15.88:0/215771219 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3178365721 0 0) 0x7fcfa4109790 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.361+0000 7fcfa8a56700 1 -- 172.21.15.88:0/215771219 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf900035b0 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfa8a56700 1 -- 172.21.15.88:0/215771219 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3152040828 0 0) 0x7fcf900035b0 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfa8a56700 1 -- 172.21.15.88:0/215771219 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa410a970 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfa8a56700 1 -- 172.21.15.88:0/215771219 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcfa00018c0 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfa8a56700 1 -- 172.21.15.88:0/215771219 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fcfa0001bf0 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfa8a56700 1 -- 172.21.15.88:0/215771219 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcfa0005ab0 con 0x7fcfa4105360 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfabcbc700 1 -- 172.21.15.88:0/215771219 >> v1:172.21.15.88:6789/0 conn(0x7fcfa4105360 legacy=0x7fcfa4105780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfabcbc700 1 -- 172.21.15.88:0/215771219 shutdown_connections 2024-03-26T21:16:15.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.362+0000 7fcfabcbc700 1 -- 172.21.15.88:0/215771219 >> 172.21.15.88:0/215771219 conn(0x7fcfa41008c0 msgr2=0x7fcfa4102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcfabcbc700 1 -- 172.21.15.88:0/215771219 shutdown_connections 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcfabcbc700 1 -- 172.21.15.88:0/215771219 wait complete. 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcfabcbc700 1 Processor -- start 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcfabcbc700 1 -- start start 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcfabcbc700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcfa41a0050 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcfa9a58700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcfa4105360 0x7fcfa41027c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42612/0 (socket says 172.21.15.88:42612) 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcfa9a58700 1 -- 172.21.15.88:0/4169767790 learned_addr learned my addr 172.21.15.88:0/4169767790 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.363+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 334970674 0 0) 0x7fcfa41a0050 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf90005530 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 369435704 0 0) 0x7fcf90005530 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfa41a0050 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcfa0001130 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2123531480 0 0) 0x7fcfa41a0050 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa41a1230 con 0x7fcfa4105360 2024-03-26T21:16:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa41a0280 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcfa41a07c0 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fcfa0001bf0 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.364+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcfa00060e0 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.365+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) ==== 46431+0+0 (unknown 2967675786 0 0) 0x7fcfa00117f0 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.365+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7fcfa0047de0 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.365+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf88005150 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.365+0000 7fcfa9257700 1 -- 172.21.15.88:0/4169767790 >> v1:172.21.15.88:6800/221932617 conn(0x7fcf9003a6e0 legacy=0x7fcf9003cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/221932617 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.370+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fcfa0017720 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.566+0000 7fcfa9257700 1 -- 172.21.15.88:0/4169767790 >> v1:172.21.15.88:6800/221932617 conn(0x7fcf9003a6e0 legacy=0x7fcf9003cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/221932617 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.637+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fcf88005c50 con 0x7fcfa4105360 2024-03-26T21:16:15.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.637+0000 7fcf9affd700 1 -- 172.21.15.88:0/4169767790 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7fcfa0017020 con 0x7fcfa4105360 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.639+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 >> v1:172.21.15.88:6800/221932617 conn(0x7fcf9003a6e0 legacy=0x7fcf9003cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.639+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 >> v1:172.21.15.88:6789/0 conn(0x7fcfa4105360 legacy=0x7fcfa41027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.639+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 shutdown_connections 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.639+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 >> 172.21.15.88:0/4169767790 conn(0x7fcfa41008c0 msgr2=0x7fcfa4078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.639+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 shutdown_connections 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:14.639+0000 7fcfabcbc700 1 -- 172.21.15.88:0/4169767790 wait complete. 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for the mgr to restart... 2024-03-26T21:16:15.676 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr epoch 5... 2024-03-26T21:16:15.793 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:15 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:15.507+0000 7f8a22548200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:16:15.793 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:15 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:15.686+0000 7f8a22548200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:16:16.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:16 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:16.662+0000 7f8a22548200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:16:16.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:16 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:16.790+0000 7f8a22548200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:16:17.397 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:17 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:17.122+0000 7f8a22548200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:16:17.397 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:17 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:17.210+0000 7f8a22548200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:16:17.397 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:17 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:17.403+0000 7f8a22548200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:16:17.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:17 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:17.587+0000 7f8a22548200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:16:18.120 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:17 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:17.868+0000 7f8a22548200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:16:18.120 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:18 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:18.126+0000 7f8a22548200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:16:18.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:18 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:18.362+0000 7f8a22548200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:16:19.485 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:19 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:19.153+0000 7f8a22548200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:16:19.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:19 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:19.718+0000 7f8a22548200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:16:19.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:19 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:19.815+0000 7f8a22548200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:20.029+0000 7f8a22548200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: Active manager daemon y restarted 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: Activating manager daemon y 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: mgrmap e6: y(active, starting, since 0.00339023s) 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:16:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:16:20.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:16:20.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:16:20.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: Manager daemon y is now available 2024-03-26T21:16:20.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:20.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:20.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:20.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:20 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:21.344 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:21 smithi088 ceph-mon[30075]: Found migration_current of "None". Setting to last migration. 2024-03-26T21:16:21.344 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:21 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:16:21.344 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:21 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:16:21.344 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:21 smithi088 ceph-mon[30075]: mgrmap e7: y(active, since 1.00595s) 2024-03-26T21:16:21.638 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.141+0000 7f3690d8d700 1 Processor -- start 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.141+0000 7f3690d8d700 1 -- start start 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.141+0000 7f3690d8d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f368c07a710 con 0x7f368c106f50 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.141+0000 7f368a59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f368c106f50 0x7f368c109340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42628/0 (socket says 172.21.15.88:42628) 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.141+0000 7f368a59c700 1 -- 172.21.15.88:0/3193507034 learned_addr learned my addr 172.21.15.88:0/3193507034 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.141+0000 7f368959a700 1 -- 172.21.15.88:0/3193507034 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077066678 0 0) 0x7f368c07a710 con 0x7f368c106f50 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.141+0000 7f368959a700 1 -- 172.21.15.88:0/3193507034 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36740035b0 con 0x7f368c106f50 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.142+0000 7f368959a700 1 -- 172.21.15.88:0/3193507034 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 20051457 0 0) 0x7f36740035b0 con 0x7f368c106f50 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.142+0000 7f368959a700 1 -- 172.21.15.88:0/3193507034 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368c10ba70 con 0x7f368c106f50 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.142+0000 7f368959a700 1 -- 172.21.15.88:0/3193507034 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f367c001580 con 0x7f368c106f50 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.142+0000 7f368959a700 1 -- 172.21.15.88:0/3193507034 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f367c001c50 con 0x7f368c106f50 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.142+0000 7f368959a700 1 -- 172.21.15.88:0/3193507034 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f367c005960 con 0x7f368c106f50 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.142+0000 7f3690d8d700 1 -- 172.21.15.88:0/3193507034 >> v1:172.21.15.88:6789/0 conn(0x7f368c106f50 legacy=0x7f368c109340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.143+0000 7f3690d8d700 1 -- 172.21.15.88:0/3193507034 shutdown_connections 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.143+0000 7f3690d8d700 1 -- 172.21.15.88:0/3193507034 >> 172.21.15.88:0/3193507034 conn(0x7f368c100960 msgr2=0x7f368c102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.143+0000 7f3690d8d700 1 -- 172.21.15.88:0/3193507034 shutdown_connections 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.143+0000 7f3690d8d700 1 -- 172.21.15.88:0/3193507034 wait complete. 2024-03-26T21:16:21.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.143+0000 7f3690d8d700 1 Processor -- start 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.143+0000 7f3690d8d700 1 -- start start 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.143+0000 7f3690d8d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f368c19a800 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f368a59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f368c106f50 0x7f368c19a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42644/0 (socket says 172.21.15.88:42644) 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f368a59c700 1 -- 172.21.15.88:0/1673320166 learned_addr learned my addr 172.21.15.88:0/1673320166 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3843030041 0 0) 0x7f368c19a800 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3674005580 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 555860015 0 0) 0x7f3674005580 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f368c19a800 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f367c001080 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2069622111 0 0) 0x7f368c19a800 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368c19a9d0 con 0x7f368c106f50 2024-03-26T21:16:21.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f368c19acc0 con 0x7f368c106f50 2024-03-26T21:16:21.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f368c19e810 con 0x7f368c106f50 2024-03-26T21:16:21.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f367c0049d0 con 0x7f368c106f50 2024-03-26T21:16:21.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.144+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f367c0062f0 con 0x7f368c106f50 2024-03-26T21:16:21.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.145+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) ==== 46431+0+0 (unknown 2967675786 0 0) 0x7f367c011a20 con 0x7f368c106f50 2024-03-26T21:16:21.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.145+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6800/221932617 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f367c001080 con 0x7f367403a7e0 2024-03-26T21:16:21.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.145+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1910167797 0 0) 0x7f367c047fa0 con 0x7f368c106f50 2024-03-26T21:16:21.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.145+0000 7f3689d9b700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6800/221932617 conn(0x7f367403a7e0 legacy=0x7f367403cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/221932617 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.346+0000 7f3689d9b700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6800/221932617 conn(0x7f367403a7e0 legacy=0x7f367403cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/221932617 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:16.747+0000 7f3689d9b700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6800/221932617 conn(0x7f367403a7e0 legacy=0x7f367403cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/221932617 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:17.548+0000 7f3689d9b700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6800/221932617 conn(0x7f367403a7e0 legacy=0x7f367403cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/221932617 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:19.149+0000 7f3689d9b700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6800/221932617 conn(0x7f367403a7e0 legacy=0x7f367403cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/221932617 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:20.044+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mgrmap(e 6) ==== 46233+0+0 (unknown 3563312256 0 0) 0x7f367c005bf0 con 0x7f368c106f50 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:20.044+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6800/221932617 conn(0x7f367403a7e0 legacy=0x7f367403cca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.044+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mgrmap(e 7) ==== 46325+0+0 (unknown 622173267 0 0) 0x7f367c049210 con 0x7f368c106f50 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.044+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6800/3808082170 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f367c001080 con 0x7f367403efb0 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.046+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7f367c001080 con 0x7f367403efb0 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.052+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 --> v1:172.21.15.88:6800/3808082170 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f368c10b5b0 con 0x7f367403efb0 2024-03-26T21:16:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.052+0000 7f367b7fe700 1 -- 172.21.15.88:0/1673320166 <== mgr.14118 v1:172.21.15.88:6800/3808082170 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (unknown 0 0 96372106) 0x7f368c10b5b0 con 0x7f367403efb0 2024-03-26T21:16:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.052+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6800/3808082170 conn(0x7f367403efb0 legacy=0x7f36740413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.052+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 >> v1:172.21.15.88:6789/0 conn(0x7f368c106f50 legacy=0x7f368c19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.053+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 shutdown_connections 2024-03-26T21:16:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.053+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 >> 172.21.15.88:0/1673320166 conn(0x7f368c100960 msgr2=0x7f368c190ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.053+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 shutdown_connections 2024-03-26T21:16:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:21.053+0000 7f3690d8d700 1 -- 172.21.15.88:0/1673320166 wait complete. 2024-03-26T21:16:21.644 INFO:teuthology.orchestra.run.smithi088.stdout:mgr epoch 5 is available 2024-03-26T21:16:21.645 INFO:teuthology.orchestra.run.smithi088.stdout:Setting orchestrator backend to cephadm... 2024-03-26T21:16:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:22 smithi088 ceph-mon[30075]: from='client.14122 v1:172.21.15.88:0/1673320166' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T21:16:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:22 smithi088 ceph-mon[30075]: from='client.14122 v1:172.21.15.88:0/1673320166' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T21:16:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:22 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:22 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:22 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:22.941 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.089+0000 7f498b16a700 1 Processor -- start 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.089+0000 7f498b16a700 1 -- start start 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.089+0000 7f498b16a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f498407a710 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.089+0000 7f4988f06700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4984106f00 0x7f49841092f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42716/0 (socket says 172.21.15.88:42716) 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.089+0000 7f4988f06700 1 -- 172.21.15.88:0/427693797 learned_addr learned my addr 172.21.15.88:0/427693797 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.089+0000 7f49837fe700 1 -- 172.21.15.88:0/427693797 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231717529 0 0) 0x7f498407a710 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f49837fe700 1 -- 172.21.15.88:0/427693797 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49740035b0 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f49837fe700 1 -- 172.21.15.88:0/427693797 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2800938936 0 0) 0x7f49740035b0 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f49837fe700 1 -- 172.21.15.88:0/427693797 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498410ba20 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f49837fe700 1 -- 172.21.15.88:0/427693797 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f4978001580 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f49837fe700 1 -- 172.21.15.88:0/427693797 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f4978001d20 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f49837fe700 1 -- 172.21.15.88:0/427693797 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f4978005960 con 0x7f4984106f00 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f498b16a700 1 -- 172.21.15.88:0/427693797 >> v1:172.21.15.88:6789/0 conn(0x7f4984106f00 legacy=0x7f49841092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:22.942 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f498b16a700 1 -- 172.21.15.88:0/427693797 shutdown_connections 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f498b16a700 1 -- 172.21.15.88:0/427693797 >> 172.21.15.88:0/427693797 conn(0x7f49841008d0 msgr2=0x7f4984102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f498b16a700 1 -- 172.21.15.88:0/427693797 shutdown_connections 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.090+0000 7f498b16a700 1 -- 172.21.15.88:0/427693797 wait complete. 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.091+0000 7f498b16a700 1 Processor -- start 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.091+0000 7f498b16a700 1 -- start start 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.091+0000 7f498b16a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f498419a7a0 con 0x7f4984106f00 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.091+0000 7f4988f06700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4984106f00 0x7f498419a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42728/0 (socket says 172.21.15.88:42728) 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.091+0000 7f4988f06700 1 -- 172.21.15.88:0/988127981 learned_addr learned my addr 172.21.15.88:0/988127981 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.091+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2962214028 0 0) 0x7f498419a7a0 con 0x7f4984106f00 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.091+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4974005530 con 0x7f4984106f00 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3342437621 0 0) 0x7f4974005530 con 0x7f4984106f00 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f498419a7a0 con 0x7f4984106f00 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f4978001080 con 0x7f4984106f00 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2397718638 0 0) 0x7f498419a7a0 con 0x7f4984106f00 2024-03-26T21:16:22.943 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498419a970 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f498419ac60 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f4978004ab0 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f49780062e0 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.092+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f498419e7b0 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.093+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 7) ==== 46325+0+0 (unknown 622173267 0 0) 0x7f4978011ad0 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.093+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4984063c70 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.093+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f4978047af0 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.099+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f4978017640 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.131+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f4978046660 con 0x7f4984106f00 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.318+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 --> v1:172.21.15.88:6800/3808082170 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f498402df20 con 0x7f497403a6d0 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.326+0000 7f4981ffb700 1 -- 172.21.15.88:0/988127981 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f498402df20 con 0x7f497403a6d0 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.329+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 >> v1:172.21.15.88:6800/3808082170 conn(0x7f497403a6d0 legacy=0x7f497403cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.329+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 >> v1:172.21.15.88:6789/0 conn(0x7f4984106f00 legacy=0x7f498419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:22.944 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.329+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 shutdown_connections 2024-03-26T21:16:22.945 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.329+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 >> 172.21.15.88:0/988127981 conn(0x7f49841008d0 msgr2=0x7f4984190d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:22.945 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.329+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 shutdown_connections 2024-03-26T21:16:22.945 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:22.329+0000 7f498b16a700 1 -- 172.21.15.88:0/988127981 wait complete. 2024-03-26T21:16:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:23 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:21] ENGINE Bus STARTING 2024-03-26T21:16:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:23 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:21] ENGINE Serving on https://172.21.15.88:7150 2024-03-26T21:16:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:23 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:21] ENGINE Serving on http://172.21.15.88:8765 2024-03-26T21:16:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:23 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:21] ENGINE Bus STARTED 2024-03-26T21:16:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:23 smithi088 ceph-mon[30075]: mgrmap e8: y(active, since 2s) 2024-03-26T21:16:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:23 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:23 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:24.166 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-mon[30075]: from='client.14130 v1:172.21.15.88:0/988127981' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.394+0000 7f80e77f1700 1 Processor -- start 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.394+0000 7f80e77f1700 1 -- start start 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.394+0000 7f80e77f1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80e0109780 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.394+0000 7f80e558d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f80e0105350 0x7f80e0105770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42744/0 (socket says 172.21.15.88:42744) 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.394+0000 7f80e558d700 1 -- 172.21.15.88:0/2252438353 learned_addr learned my addr 172.21.15.88:0/2252438353 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.394+0000 7f80d7fff700 1 -- 172.21.15.88:0/2252438353 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147231521 0 0) 0x7f80e0109780 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.394+0000 7f80d7fff700 1 -- 172.21.15.88:0/2252438353 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80cc0035b0 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80d7fff700 1 -- 172.21.15.88:0/2252438353 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2371429238 0 0) 0x7f80cc0035b0 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80d7fff700 1 -- 172.21.15.88:0/2252438353 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e010a960 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80d7fff700 1 -- 172.21.15.88:0/2252438353 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f80dc001580 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80d7fff700 1 -- 172.21.15.88:0/2252438353 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f80dc001d20 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80d7fff700 1 -- 172.21.15.88:0/2252438353 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f80dc005960 con 0x7f80e0105350 2024-03-26T21:16:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80e77f1700 1 -- 172.21.15.88:0/2252438353 >> v1:172.21.15.88:6789/0 conn(0x7f80e0105350 legacy=0x7f80e0105770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80e77f1700 1 -- 172.21.15.88:0/2252438353 shutdown_connections 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80e77f1700 1 -- 172.21.15.88:0/2252438353 >> 172.21.15.88:0/2252438353 conn(0x7f80e01008d0 msgr2=0x7f80e0102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80e77f1700 1 -- 172.21.15.88:0/2252438353 shutdown_connections 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.395+0000 7f80e77f1700 1 -- 172.21.15.88:0/2252438353 wait complete. 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.396+0000 7f80e77f1700 1 Processor -- start 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.396+0000 7f80e77f1700 1 -- start start 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.396+0000 7f80e77f1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80e019a800 con 0x7f80e0105350 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.396+0000 7f80e558d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f80e0105350 0x7f80e019a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42754/0 (socket says 172.21.15.88:42754) 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.396+0000 7f80e558d700 1 -- 172.21.15.88:0/2001034594 learned_addr learned my addr 172.21.15.88:0/2001034594 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.396+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2540922126 0 0) 0x7f80e019a800 con 0x7f80e0105350 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.396+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80cc005530 con 0x7f80e0105350 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3973921827 0 0) 0x7f80cc005530 con 0x7f80e0105350 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80e019a800 con 0x7f80e0105350 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f80dc004870 con 0x7f80e0105350 2024-03-26T21:16:24.174 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 710397778 0 0) 0x7f80e019a800 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e019a9d0 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f80e019acc0 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f80e019e810 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f80dc004a40 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f80dc006050 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.397+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f80dc006280 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.398+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c4005120 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.398+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f80dc047d70 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.403+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f80dc017640 con 0x7f80e0105350 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.603+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 --> v1:172.21.15.88:6800/3808082170 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f80c4000b90 con 0x7f80cc03a6c0 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.603+0000 7f80d67fc700 1 -- 172.21.15.88:0/2001034594 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (unknown 0 0 2070689548) 0x7f80c4000b90 con 0x7f80cc03a6c0 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.606+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 >> v1:172.21.15.88:6800/3808082170 conn(0x7f80cc03a6c0 legacy=0x7f80cc03cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.606+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 >> v1:172.21.15.88:6789/0 conn(0x7f80e0105350 legacy=0x7f80e019a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.606+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 shutdown_connections 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.606+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 >> 172.21.15.88:0/2001034594 conn(0x7f80e01008d0 msgr2=0x7f80e00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:24.175 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.606+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 shutdown_connections 2024-03-26T21:16:24.176 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:23.606+0000 7f80e77f1700 1 -- 172.21.15.88:0/2001034594 wait complete. 2024-03-26T21:16:24.176 INFO:teuthology.orchestra.run.smithi088.stdout:Generating ssh key... 2024-03-26T21:16:25.127 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: Generating public/private rsa key pair. 2024-03-26T21:16:25.127 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: Your identification has been saved in /tmp/tmp7ieg6170/key. 2024-03-26T21:16:25.127 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: Your public key has been saved in /tmp/tmp7ieg6170/key.pub. 2024-03-26T21:16:25.127 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: The key fingerprint is: 2024-03-26T21:16:25.127 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: SHA256:u8peS96kNx1GlRWq7kJxNbqxyhijucN441Dd//neYyg ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:16:25.127 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: The key's randomart image is: 2024-03-26T21:16:25.127 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: +---[RSA 3072]----+ 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | +o| 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | o+ | 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | oo. | 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | . .. +o | 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | . .S.oo+ | 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | . o oooo | 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | .o o X o= .. | 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | ..O = X+Eo..o.| 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: | o+B.=.oo.o+oo| 2024-03-26T21:16:25.128 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:24 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: +----[SHA256]-----+ 2024-03-26T21:16:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:25 smithi088 ceph-mon[30075]: from='client.14132 v1:172.21.15.88:0/2001034594' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:25 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:25 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.642+0000 7f303595c700 1 Processor -- start 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.642+0000 7f303595c700 1 -- start start 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.642+0000 7f303595c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f303007a6f0 con 0x7f3030106f20 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.642+0000 7f302effd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3030106f20 0x7f3030109310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34304/0 (socket says 172.21.15.88:34304) 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.642+0000 7f302effd700 1 -- 172.21.15.88:0/1470460293 learned_addr learned my addr 172.21.15.88:0/1470460293 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f302dffb700 1 -- 172.21.15.88:0/1470460293 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229959913 0 0) 0x7f303007a6f0 con 0x7f3030106f20 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f302dffb700 1 -- 172.21.15.88:0/1470460293 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30180035b0 con 0x7f3030106f20 2024-03-26T21:16:25.565 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f302dffb700 1 -- 172.21.15.88:0/1470460293 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2275152522 0 0) 0x7f30180035b0 con 0x7f3030106f20 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f302dffb700 1 -- 172.21.15.88:0/1470460293 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303010ba40 con 0x7f3030106f20 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f302dffb700 1 -- 172.21.15.88:0/1470460293 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3020001580 con 0x7f3030106f20 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f302dffb700 1 -- 172.21.15.88:0/1470460293 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f3020001d20 con 0x7f3030106f20 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f302dffb700 1 -- 172.21.15.88:0/1470460293 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3020005960 con 0x7f3030106f20 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.643+0000 7f303595c700 1 -- 172.21.15.88:0/1470460293 >> v1:172.21.15.88:6789/0 conn(0x7f3030106f20 legacy=0x7f3030109310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.644+0000 7f303595c700 1 -- 172.21.15.88:0/1470460293 shutdown_connections 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.644+0000 7f303595c700 1 -- 172.21.15.88:0/1470460293 >> 172.21.15.88:0/1470460293 conn(0x7f30301008d0 msgr2=0x7f3030102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.644+0000 7f303595c700 1 -- 172.21.15.88:0/1470460293 shutdown_connections 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.644+0000 7f303595c700 1 -- 172.21.15.88:0/1470460293 wait complete. 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.644+0000 7f303595c700 1 Processor -- start 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.644+0000 7f303595c700 1 -- start start 2024-03-26T21:16:25.566 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.644+0000 7f303595c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f303019a7b0 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f302effd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3030106f20 0x7f303019a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34310/0 (socket says 172.21.15.88:34310) 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f302effd700 1 -- 172.21.15.88:0/2481652335 learned_addr learned my addr 172.21.15.88:0/2481652335 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1131611373 0 0) 0x7f303019a7b0 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3018005530 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1648571475 0 0) 0x7f3018005530 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f303019a7b0 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3020001080 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 571803219 0 0) 0x7f303019a7b0 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303019a980 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f303019ac70 con 0x7f3030106f20 2024-03-26T21:16:25.567 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f303019e7c0 con 0x7f3030106f20 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f3020001c80 con 0x7f3030106f20 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.645+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3020005fa0 con 0x7f3030106f20 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.646+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f3020011770 con 0x7f3030106f20 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.646+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f301c005120 con 0x7f3030106f20 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.646+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f3020047d50 con 0x7f3030106f20 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.651+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f30200176b0 con 0x7f3030106f20 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.849+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 --> v1:172.21.15.88:6800/3808082170 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f301c000b90 con 0x7f301803a7f0 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.941+0000 7f303495a700 1 -- 172.21.15.88:0/2481652335 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f301c000b90 con 0x7f301803a7f0 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.943+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 >> v1:172.21.15.88:6800/3808082170 conn(0x7f301803a7f0 legacy=0x7f301803ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.943+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 >> v1:172.21.15.88:6789/0 conn(0x7f3030106f20 legacy=0x7f303019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.943+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 shutdown_connections 2024-03-26T21:16:25.568 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.943+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 >> 172.21.15.88:0/2481652335 conn(0x7f30301008d0 msgr2=0x7f3030190d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:25.569 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.944+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 shutdown_connections 2024-03-26T21:16:25.569 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:24.944+0000 7f303595c700 1 -- 172.21.15.88:0/2481652335 wait complete. 2024-03-26T21:16:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:26 smithi088 ceph-mon[30075]: from='client.14134 v1:172.21.15.88:0/2481652335' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:26 smithi088 ceph-mon[30075]: Generating ssh key... 2024-03-26T21:16:26.753 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUWO/Y48EDmfHaNiYlajyX6tELb0BWkW4JOsQX+RybShngd24Gy3oj4qLtRe5bHhVWkN0CRZqS0SSoAzCeTzgsr15pQfBzWWRaXDOaDzbBPqj/6lQamVv+/V1pIXaHdBUe3r9BwWFbbe+Dg2Zw1flB3szBMatBHWh//zWPDe1clCCwZm5yF2Bvdd6P7ySXyPOp/H8h+AkfmjIxzz/PKUg0mOsOchKevDTfYp2R/pS1RRMt8VB1IgHskqkwEbKKsGfe6knoMDPAwr0a7phok+M+Mh6Uu5XG53PqtMOePyeNFmT6SXBDBBcDHCZVz6F7JveUlI+XnEMWSAP8MCNaiC+FHZ+ZdcQTWTIpPJiV1clv8f2CFBEThuB7tSXcEiRrD1xhgGri6l16AsQ0cZixcW1Sm76Z+NRR1Kkr67ScDfRvooPsw5Dd8jPnFMVnBNz9IM92A2pRKeFMGgR5jf4WN1YjJeX7BnadJ4Zbuq18P3IgpYabkTyAHftfOqt3B2X5lRs= ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:16:26.753 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.027+0000 7f3e1ea35700 1 Processor -- start 2024-03-26T21:16:26.753 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.027+0000 7f3e1ea35700 1 -- start start 2024-03-26T21:16:26.753 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.027+0000 7f3e1ea35700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3e1807a6d0 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.027+0000 7f3e17fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3e18104d80 0x7f3e181051a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34326/0 (socket says 172.21.15.88:34326) 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.027+0000 7f3e17fff700 1 -- 172.21.15.88:0/2934588418 learned_addr learned my addr 172.21.15.88:0/2934588418 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.027+0000 7f3e16ffd700 1 -- 172.21.15.88:0/2934588418 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652117380 0 0) 0x7f3e1807a6d0 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.027+0000 7f3e16ffd700 1 -- 172.21.15.88:0/2934588418 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e000035b0 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e16ffd700 1 -- 172.21.15.88:0/2934588418 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4172654389 0 0) 0x7f3e000035b0 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e16ffd700 1 -- 172.21.15.88:0/2934588418 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e181068c0 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e16ffd700 1 -- 172.21.15.88:0/2934588418 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3e08001580 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e16ffd700 1 -- 172.21.15.88:0/2934588418 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f3e08001d20 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e16ffd700 1 -- 172.21.15.88:0/2934588418 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3e08005960 con 0x7f3e18104d80 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e1ea35700 1 -- 172.21.15.88:0/2934588418 >> v1:172.21.15.88:6789/0 conn(0x7f3e18104d80 legacy=0x7f3e181051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e1ea35700 1 -- 172.21.15.88:0/2934588418 shutdown_connections 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e1ea35700 1 -- 172.21.15.88:0/2934588418 >> 172.21.15.88:0/2934588418 conn(0x7f3e18100960 msgr2=0x7f3e18102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:26.754 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e1ea35700 1 -- 172.21.15.88:0/2934588418 shutdown_connections 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.028+0000 7f3e1ea35700 1 -- 172.21.15.88:0/2934588418 wait complete. 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.029+0000 7f3e1ea35700 1 Processor -- start 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.029+0000 7f3e1ea35700 1 -- start start 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.029+0000 7f3e1ea35700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3e1819c960 con 0x7f3e18104d80 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.029+0000 7f3e17fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3e18104d80 0x7f3e1819c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34340/0 (socket says 172.21.15.88:34340) 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.029+0000 7f3e17fff700 1 -- 172.21.15.88:0/1677279875 learned_addr learned my addr 172.21.15.88:0/1677279875 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 672731016 0 0) 0x7f3e1819c960 con 0x7f3e18104d80 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e00005530 con 0x7f3e18104d80 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1056878031 0 0) 0x7f3e00005530 con 0x7f3e18104d80 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e1819c960 con 0x7f3e18104d80 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3e08001080 con 0x7f3e18104d80 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1053418443 0 0) 0x7f3e1819c960 con 0x7f3e18104d80 2024-03-26T21:16:26.755 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1819cb30 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1819ce20 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3e181a0970 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f3e08001bc0 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.030+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3e08005f30 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.031+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f3e08011640 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.031+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e04005150 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.031+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f3e08047c50 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.036+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f3e08017570 con 0x7f3e18104d80 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.238+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 --> v1:172.21.15.88:6800/3808082170 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f3e04000bc0 con 0x7f3e0003a840 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.239+0000 7f3e157fa700 1 -- 172.21.15.88:0/1677279875 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (unknown 0 0 1767964011) 0x7f3e04000bc0 con 0x7f3e0003a840 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.241+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 >> v1:172.21.15.88:6800/3808082170 conn(0x7f3e0003a840 legacy=0x7f3e0003cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.241+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 >> v1:172.21.15.88:6789/0 conn(0x7f3e18104d80 legacy=0x7f3e1819c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.241+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 shutdown_connections 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.241+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 >> 172.21.15.88:0/1677279875 conn(0x7f3e18100960 msgr2=0x7f3e18192ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.241+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 shutdown_connections 2024-03-26T21:16:26.757 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:26.241+0000 7f3e1ea35700 1 -- 172.21.15.88:0/1677279875 wait complete. 2024-03-26T21:16:26.757 INFO:teuthology.orchestra.run.smithi088.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T21:16:26.757 INFO:teuthology.orchestra.run.smithi088.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T21:16:26.757 INFO:teuthology.orchestra.run.smithi088.stdout:Adding host smithi088... 2024-03-26T21:16:27.420 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:27 smithi088 ceph-mon[30075]: from='client.14136 v1:172.21.15.88:0/1677279875' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:28.392 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:28 smithi088 ceph-mon[30075]: from='client.14138 v1:172.21.15.88:0/3699609397' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "addr": "172.21.15.88", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:28.393 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:28 smithi088 ceph-mon[30075]: Deploying cephadm binary to smithi088 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Added host 'smithi088' with addr '172.21.15.88' 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f999a700 1 Processor -- start 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f999a700 1 -- start start 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f999a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27f407b360 con 0x7f27f407c760 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f2ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f27f407c760 0x7f27f407abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34350/0 (socket says 172.21.15.88:34350) 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f2ffd700 1 -- 172.21.15.88:0/1639505040 learned_addr learned my addr 172.21.15.88:0/1639505040 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f1ffb700 1 -- 172.21.15.88:0/1639505040 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157985571 0 0) 0x7f27f407b360 con 0x7f27f407c760 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f1ffb700 1 -- 172.21.15.88:0/1639505040 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27dc0035b0 con 0x7f27f407c760 2024-03-26T21:16:32.370 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f1ffb700 1 -- 172.21.15.88:0/1639505040 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1710002073 0 0) 0x7f27dc0035b0 con 0x7f27f407c760 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f1ffb700 1 -- 172.21.15.88:0/1639505040 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f407b530 con 0x7f27f407c760 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.217+0000 7f27f1ffb700 1 -- 172.21.15.88:0/1639505040 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f27e4001580 con 0x7f27f407c760 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.218+0000 7f27f1ffb700 1 -- 172.21.15.88:0/1639505040 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f27e4001d20 con 0x7f27f407c760 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.218+0000 7f27f1ffb700 1 -- 172.21.15.88:0/1639505040 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f27e4005960 con 0x7f27f407c760 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.218+0000 7f27f999a700 1 -- 172.21.15.88:0/1639505040 >> v1:172.21.15.88:6789/0 conn(0x7f27f407c760 legacy=0x7f27f407abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.218+0000 7f27f999a700 1 -- 172.21.15.88:0/1639505040 shutdown_connections 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.218+0000 7f27f999a700 1 -- 172.21.15.88:0/1639505040 >> 172.21.15.88:0/1639505040 conn(0x7f27f4101960 msgr2=0x7f27f4103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.218+0000 7f27f999a700 1 -- 172.21.15.88:0/1639505040 shutdown_connections 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.218+0000 7f27f999a700 1 -- 172.21.15.88:0/1639505040 wait complete. 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f999a700 1 Processor -- start 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f999a700 1 -- start start 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f999a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f27f419c9c0 con 0x7f27f407c760 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f2ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f27f407c760 0x7f27f419c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34360/0 (socket says 172.21.15.88:34360) 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f2ffd700 1 -- 172.21.15.88:0/3699609397 learned_addr learned my addr 172.21.15.88:0/3699609397 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:32.371 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100285532 0 0) 0x7f27f419c9c0 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27dc005530 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1197273867 0 0) 0x7f27dc005530 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27f419c9c0 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.219+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f27e4001080 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.220+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3281582229 0 0) 0x7f27f419c9c0 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.220+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f419cb90 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.220+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27f419ce80 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.220+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f27f41a09d0 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.220+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f27e4001c60 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.220+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f27e4005fa0 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.221+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f27e4011770 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.221+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e0005150 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.221+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f27e4047d10 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.226+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f27e4017690 con 0x7f27f407c760 2024-03-26T21:16:32.372 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:27.426+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 --> v1:172.21.15.88:6800/3808082170 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi088", "addr": "172.21.15.88", "target": ["mon-mgr", ""]}) -- 0x7f27e0000bc0 con 0x7f27dc03a7f0 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:31.617+0000 7f27f8998700 1 -- 172.21.15.88:0/3699609397 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (unknown 0 0 3654992492) 0x7f27e0000bc0 con 0x7f27dc03a7f0 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:31.619+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 >> v1:172.21.15.88:6800/3808082170 conn(0x7f27dc03a7f0 legacy=0x7f27dc03ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:31.619+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 >> v1:172.21.15.88:6789/0 conn(0x7f27f407c760 legacy=0x7f27f419c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:31.619+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 shutdown_connections 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:31.619+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 >> 172.21.15.88:0/3699609397 conn(0x7f27f4101960 msgr2=0x7f27f410a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:31.619+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 shutdown_connections 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:31.619+0000 7f27f999a700 1 -- 172.21.15.88:0/3699609397 wait complete. 2024-03-26T21:16:32.373 INFO:teuthology.orchestra.run.smithi088.stdout:Deploying unmanaged mon service... 2024-03-26T21:16:32.690 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:32 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:32.690 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:32 smithi088 ceph-mon[30075]: Added host smithi088 2024-03-26T21:16:32.690 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:32 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:33.628 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T21:16:33.628 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.801+0000 7f558eaa3700 1 Processor -- start 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.801+0000 7f558eaa3700 1 -- start start 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.801+0000 7f558eaa3700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f558810a920 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.801+0000 7f558c83f700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5588104760 0x7f5588104b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34370/0 (socket says 172.21.15.88:34370) 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.801+0000 7f558c83f700 1 -- 172.21.15.88:0/4117253831 learned_addr learned my addr 172.21.15.88:0/4117253831 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f55877fe700 1 -- 172.21.15.88:0/4117253831 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876693413 0 0) 0x7f558810a920 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f55877fe700 1 -- 172.21.15.88:0/4117253831 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55780035b0 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f55877fe700 1 -- 172.21.15.88:0/4117253831 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2624705707 0 0) 0x7f55780035b0 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f55877fe700 1 -- 172.21.15.88:0/4117253831 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558810bb00 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f55877fe700 1 -- 172.21.15.88:0/4117253831 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f557c0018c0 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f55877fe700 1 -- 172.21.15.88:0/4117253831 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f557c001bf0 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f55877fe700 1 -- 172.21.15.88:0/4117253831 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f557c005b20 con 0x7f5588104760 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f558eaa3700 1 -- 172.21.15.88:0/4117253831 >> v1:172.21.15.88:6789/0 conn(0x7f5588104760 legacy=0x7f5588104b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f558eaa3700 1 -- 172.21.15.88:0/4117253831 shutdown_connections 2024-03-26T21:16:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.802+0000 7f558eaa3700 1 -- 172.21.15.88:0/4117253831 >> 172.21.15.88:0/4117253831 conn(0x7f5588100050 msgr2=0x7f55881024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f558eaa3700 1 -- 172.21.15.88:0/4117253831 shutdown_connections 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f558eaa3700 1 -- 172.21.15.88:0/4117253831 wait complete. 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f558eaa3700 1 Processor -- start 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f558eaa3700 1 -- start start 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f558eaa3700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f558819a790 con 0x7f5588104760 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f558c83f700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5588104760 0x7f558819a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34372/0 (socket says 172.21.15.88:34372) 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f558c83f700 1 -- 172.21.15.88:0/1435031338 learned_addr learned my addr 172.21.15.88:0/1435031338 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873816656 0 0) 0x7f558819a790 con 0x7f5588104760 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.803+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5578005530 con 0x7f5588104760 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2217209505 0 0) 0x7f5578005530 con 0x7f5588104760 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f558819a790 con 0x7f5588104760 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f557c001130 con 0x7f5588104760 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2354858148 0 0) 0x7f558819a790 con 0x7f5588104760 2024-03-26T21:16:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558819a960 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f558819ac50 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7f557c003930 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f557c0064c0 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.804+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f558819e7a0 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.805+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f557c006660 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.805+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f557c047f40 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.805+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5588063c70 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:32.810+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f557c0177f0 con 0x7f5588104760 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.022+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 --> v1:172.21.15.88:6800/3808082170 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5588062a80 con 0x7f557803a6a0 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.030+0000 7f5585ffb700 1 -- 172.21.15.88:0/1435031338 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7f5588062a80 con 0x7f557803a6a0 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.032+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 >> v1:172.21.15.88:6800/3808082170 conn(0x7f557803a6a0 legacy=0x7f557803cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.032+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 >> v1:172.21.15.88:6789/0 conn(0x7f5588104760 legacy=0x7f558819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.032+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 shutdown_connections 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.032+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 >> 172.21.15.88:0/1435031338 conn(0x7f5588100050 msgr2=0x7f55881024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.033+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 shutdown_connections 2024-03-26T21:16:33.632 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:33.033+0000 7f558eaa3700 1 -- 172.21.15.88:0/1435031338 wait complete. 2024-03-26T21:16:33.632 INFO:teuthology.orchestra.run.smithi088.stdout:Deploying unmanaged mgr service... 2024-03-26T21:16:34.208 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:34 smithi088 ceph-mon[30075]: from='client.14140 v1:172.21.15.88:0/1435031338' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:34.209 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:34 smithi088 ceph-mon[30075]: Saving service mon spec with placement count:5 2024-03-26T21:16:34.209 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:34 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:35.709 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:35 smithi088 ceph-mon[30075]: from='client.14142 v1:172.21.15.88:0/1804288072' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:35.709 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:35 smithi088 ceph-mon[30075]: Saving service mgr spec with placement count:2 2024-03-26T21:16:35.709 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:35 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:35.716 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T21:16:35.716 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.066+0000 7fbc2eecd700 1 Processor -- start 2024-03-26T21:16:35.716 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.066+0000 7fbc2eecd700 1 -- start start 2024-03-26T21:16:35.716 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.066+0000 7fbc2eecd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc2810a8c0 con 0x7fbc28106490 2024-03-26T21:16:35.716 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.066+0000 7fbc2cc69700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbc28106490 0x7fbc281068b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54938/0 (socket says 172.21.15.88:54938) 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.066+0000 7fbc2cc69700 1 -- 172.21.15.88:0/334902536 learned_addr learned my addr 172.21.15.88:0/334902536 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.066+0000 7fbc277fe700 1 -- 172.21.15.88:0/334902536 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777831167 0 0) 0x7fbc2810a8c0 con 0x7fbc28106490 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc277fe700 1 -- 172.21.15.88:0/334902536 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc180035b0 con 0x7fbc28106490 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc277fe700 1 -- 172.21.15.88:0/334902536 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2785253671 0 0) 0x7fbc180035b0 con 0x7fbc28106490 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc277fe700 1 -- 172.21.15.88:0/334902536 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2810baa0 con 0x7fbc28106490 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc277fe700 1 -- 172.21.15.88:0/334902536 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fbc1c001580 con 0x7fbc28106490 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc277fe700 1 -- 172.21.15.88:0/334902536 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fbc1c001d20 con 0x7fbc28106490 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc277fe700 1 -- 172.21.15.88:0/334902536 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fbc1c005960 con 0x7fbc28106490 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc2eecd700 1 -- 172.21.15.88:0/334902536 >> v1:172.21.15.88:6789/0 conn(0x7fbc28106490 legacy=0x7fbc281068b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc2eecd700 1 -- 172.21.15.88:0/334902536 shutdown_connections 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.067+0000 7fbc2eecd700 1 -- 172.21.15.88:0/334902536 >> 172.21.15.88:0/334902536 conn(0x7fbc28101a30 msgr2=0x7fbc28103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.068+0000 7fbc2eecd700 1 -- 172.21.15.88:0/334902536 shutdown_connections 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.068+0000 7fbc2eecd700 1 -- 172.21.15.88:0/334902536 wait complete. 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.068+0000 7fbc2eecd700 1 Processor -- start 2024-03-26T21:16:35.717 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.068+0000 7fbc2eecd700 1 -- start start 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.068+0000 7fbc2eecd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc2819a720 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.068+0000 7fbc2cc69700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbc28106490 0x7fbc2819a010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54942/0 (socket says 172.21.15.88:54942) 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.068+0000 7fbc2cc69700 1 -- 172.21.15.88:0/1804288072 learned_addr learned my addr 172.21.15.88:0/1804288072 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2342880248 0 0) 0x7fbc2819a720 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc18005530 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2750223743 0 0) 0x7fbc18005530 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc2819a720 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fbc1c004870 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3011198300 0 0) 0x7fbc2819a720 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2819a8f0 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2819abe0 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 1290344078 0 0) 0x7fbc1c004a40 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fbc1c0061c0 con 0x7fbc28106490 2024-03-26T21:16:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.069+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc2819e730 con 0x7fbc28106490 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.070+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7fbc1c011a40 con 0x7fbc28106490 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.070+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7fbc1c047d90 con 0x7fbc28106490 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.070+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2810b600 con 0x7fbc28106490 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.075+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fbc1c0175b0 con 0x7fbc28106490 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.292+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 --> v1:172.21.15.88:6800/3808082170 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fbc28062a80 con 0x7fbc1803a6a0 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.299+0000 7fbc25ffb700 1 -- 172.21.15.88:0/1804288072 <== mgr.14118 v1:172.21.15.88:6800/3808082170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7fbc28062a80 con 0x7fbc1803a6a0 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.301+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 >> v1:172.21.15.88:6800/3808082170 conn(0x7fbc1803a6a0 legacy=0x7fbc1803cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.301+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 >> v1:172.21.15.88:6789/0 conn(0x7fbc28106490 legacy=0x7fbc2819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.301+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 shutdown_connections 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.301+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 >> 172.21.15.88:0/1804288072 conn(0x7fbc28101a30 msgr2=0x7fbc28103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.301+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 shutdown_connections 2024-03-26T21:16:35.719 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:34.301+0000 7fbc2eecd700 1 -- 172.21.15.88:0/1804288072 wait complete. 2024-03-26T21:16:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:36 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:36 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.200+0000 7f49d495e700 1 Processor -- start 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.200+0000 7f49d495e700 1 -- start start 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49d495e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49d010a930 con 0x7f49d0104720 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49ce59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f49d0104720 0x7f49d0104b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54952/0 (socket says 172.21.15.88:54952) 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49ce59c700 1 -- 172.21.15.88:0/559299703 learned_addr learned my addr 172.21.15.88:0/559299703 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49cd59a700 1 -- 172.21.15.88:0/559299703 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964721639 0 0) 0x7f49d010a930 con 0x7f49d0104720 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49cd59a700 1 -- 172.21.15.88:0/559299703 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b80035b0 con 0x7f49d0104720 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49cd59a700 1 -- 172.21.15.88:0/559299703 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2590086283 0 0) 0x7f49b80035b0 con 0x7f49d0104720 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49cd59a700 1 -- 172.21.15.88:0/559299703 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d010bb10 con 0x7f49d0104720 2024-03-26T21:16:37.090 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49cd59a700 1 -- 172.21.15.88:0/559299703 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f49c00018c0 con 0x7f49d0104720 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49cd59a700 1 -- 172.21.15.88:0/559299703 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f49c0001bf0 con 0x7f49d0104720 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.201+0000 7f49cd59a700 1 -- 172.21.15.88:0/559299703 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f49c0005b70 con 0x7f49d0104720 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.202+0000 7f49d495e700 1 -- 172.21.15.88:0/559299703 >> v1:172.21.15.88:6789/0 conn(0x7f49d0104720 legacy=0x7f49d0104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.202+0000 7f49d495e700 1 -- 172.21.15.88:0/559299703 shutdown_connections 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.202+0000 7f49d495e700 1 -- 172.21.15.88:0/559299703 >> 172.21.15.88:0/559299703 conn(0x7f49d0100040 msgr2=0x7f49d01024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.202+0000 7f49d495e700 1 -- 172.21.15.88:0/559299703 shutdown_connections 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.202+0000 7f49d495e700 1 -- 172.21.15.88:0/559299703 wait complete. 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.203+0000 7f49d495e700 1 Processor -- start 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.203+0000 7f49d495e700 1 -- start start 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.203+0000 7f49d495e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49d01a0070 con 0x7f49d0104720 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.203+0000 7f49ce59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f49d0104720 0x7f49d0106f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54964/0 (socket says 172.21.15.88:54964) 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.203+0000 7f49ce59c700 1 -- 172.21.15.88:0/1145244595 learned_addr learned my addr 172.21.15.88:0/1145244595 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.203+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3879618596 0 0) 0x7f49d01a0070 con 0x7f49d0104720 2024-03-26T21:16:37.091 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.203+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b8005530 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2827840519 0 0) 0x7f49b8005530 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49d01a0070 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f49c0001130 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1097881589 0 0) 0x7f49d01a0070 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d01a1250 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49d01a02a0 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f49d01a07e0 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f49c0003bb0 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.204+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f49c00065a0 con 0x7f49d0104720 2024-03-26T21:16:37.092 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.205+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f49c0011cd0 con 0x7f49d0104720 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.205+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b0005150 con 0x7f49d0104720 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.205+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f49c0047fe0 con 0x7f49d0104720 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.210+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f49c00178e0 con 0x7f49d0104720 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.442+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 --> v1:172.21.15.88:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f49b0005440 con 0x7f49d0104720 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.444+0000 7f49bf7fe700 1 -- 172.21.15.88:0/1145244595 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (unknown 2684808024 0 0) 0x7f49c00171e0 con 0x7f49d0104720 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.446+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 >> v1:172.21.15.88:6800/3808082170 conn(0x7f49b803a6a0 legacy=0x7f49b803cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.446+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 >> v1:172.21.15.88:6789/0 conn(0x7f49d0104720 legacy=0x7f49d0106f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.446+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 shutdown_connections 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.446+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 >> 172.21.15.88:0/1145244595 conn(0x7f49d0100040 msgr2=0x7f49d01024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.447+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 shutdown_connections 2024-03-26T21:16:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:36.447+0000 7f49d495e700 1 -- 172.21.15.88:0/1145244595 wait complete. 2024-03-26T21:16:37.721 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1145244595' entity='client.admin' 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.546+0000 7f51a9428700 1 Processor -- start 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.546+0000 7f51a9428700 1 -- start start 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.546+0000 7f51a9428700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f51a410a920 con 0x7f51a4104760 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.546+0000 7f51a2ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f51a4104760 0x7f51a4104b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54974/0 (socket says 172.21.15.88:54974) 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.546+0000 7f51a2ffd700 1 -- 172.21.15.88:0/371605177 learned_addr learned my addr 172.21.15.88:0/371605177 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.546+0000 7f51a1ffb700 1 -- 172.21.15.88:0/371605177 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060929404 0 0) 0x7f51a410a920 con 0x7f51a4104760 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.546+0000 7f51a1ffb700 1 -- 172.21.15.88:0/371605177 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f518c0035b0 con 0x7f51a4104760 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a1ffb700 1 -- 172.21.15.88:0/371605177 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 723762848 0 0) 0x7f518c0035b0 con 0x7f51a4104760 2024-03-26T21:16:38.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a1ffb700 1 -- 172.21.15.88:0/371605177 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a410bb00 con 0x7f51a4104760 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a1ffb700 1 -- 172.21.15.88:0/371605177 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f51940018c0 con 0x7f51a4104760 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a1ffb700 1 -- 172.21.15.88:0/371605177 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f5194001bf0 con 0x7f51a4104760 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a1ffb700 1 -- 172.21.15.88:0/371605177 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f5194005b70 con 0x7f51a4104760 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a9428700 1 -- 172.21.15.88:0/371605177 >> v1:172.21.15.88:6789/0 conn(0x7f51a4104760 legacy=0x7f51a4104b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a9428700 1 -- 172.21.15.88:0/371605177 shutdown_connections 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a9428700 1 -- 172.21.15.88:0/371605177 >> 172.21.15.88:0/371605177 conn(0x7f51a4100050 msgr2=0x7f51a41024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a9428700 1 -- 172.21.15.88:0/371605177 shutdown_connections 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.547+0000 7f51a9428700 1 -- 172.21.15.88:0/371605177 wait complete. 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.548+0000 7f51a9428700 1 Processor -- start 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.548+0000 7f51a9428700 1 -- start start 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.548+0000 7f51a9428700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f51a419a7b0 con 0x7f51a4104760 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.548+0000 7f51a2ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f51a4104760 0x7f51a419a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54980/0 (socket says 172.21.15.88:54980) 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.548+0000 7f51a2ffd700 1 -- 172.21.15.88:0/1542753856 learned_addr learned my addr 172.21.15.88:0/1542753856 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578498451 0 0) 0x7f51a419a7b0 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f518c005530 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4187100652 0 0) 0x7f518c005530 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51a419a7b0 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f5194001130 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1990090430 0 0) 0x7f51a419a7b0 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a419a980 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f51a419ac70 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f51a419e7c0 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f5194001500 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.549+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f5194006440 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.550+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7f5194011c30 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.550+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5190005150 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.550+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7f5194048220 con 0x7f51a4104760 2024-03-26T21:16:38.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.555+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f5194017b70 con 0x7f51a4104760 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.757+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 --> v1:172.21.15.88:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f5190005440 con 0x7f51a4104760 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.760+0000 7f518bfff700 1 -- 172.21.15.88:0/1542753856 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (unknown 1496729260 0 0) 0x7f5194017470 con 0x7f51a4104760 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.762+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 >> v1:172.21.15.88:6800/3808082170 conn(0x7f518c03a600 legacy=0x7f518c03cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.762+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 >> v1:172.21.15.88:6789/0 conn(0x7f51a4104760 legacy=0x7f51a419a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.762+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 shutdown_connections 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.762+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 >> 172.21.15.88:0/1542753856 conn(0x7f51a4100050 msgr2=0x7f51a41024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.762+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 shutdown_connections 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:37.762+0000 7f51a9428700 1 -- 172.21.15.88:0/1542753856 wait complete. 2024-03-26T21:16:38.418 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling the dashboard module... 2024-03-26T21:16:38.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1542753856' entity='client.admin' 2024-03-26T21:16:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:40 smithi088 ceph-mon[30075]: from='mgr.14118 v1:172.21.15.88:0/64777244' entity='mgr.y' 2024-03-26T21:16:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/762659805' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T21:16:40.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:40 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ignoring --setuser ceph since I am not root 2024-03-26T21:16:40.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:40 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ignoring --setgroup ceph since I am not root 2024-03-26T21:16:40.840 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:40 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:40.754+0000 7f7d65f36200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:16:40.840 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:40 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:40.846+0000 7f7d65f36200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:16:40.844 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.079+0000 7fcd8993b700 1 Processor -- start 2024-03-26T21:16:40.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.079+0000 7fcd8993b700 1 -- start start 2024-03-26T21:16:40.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.079+0000 7fcd8993b700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd8407a710 con 0x7fcd84106f20 2024-03-26T21:16:40.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.079+0000 7fcd82ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcd84106f20 0x7fcd84109310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54982/0 (socket says 172.21.15.88:54982) 2024-03-26T21:16:40.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.079+0000 7fcd82ffd700 1 -- 172.21.15.88:0/3583496929 learned_addr learned my addr 172.21.15.88:0/3583496929 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:40.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.080+0000 7fcd827fc700 1 -- 172.21.15.88:0/3583496929 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1261209319 0 0) 0x7fcd8407a710 con 0x7fcd84106f20 2024-03-26T21:16:40.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.080+0000 7fcd827fc700 1 -- 172.21.15.88:0/3583496929 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd700035b0 con 0x7fcd84106f20 2024-03-26T21:16:40.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.080+0000 7fcd827fc700 1 -- 172.21.15.88:0/3583496929 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4196589706 0 0) 0x7fcd700035b0 con 0x7fcd84106f20 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.080+0000 7fcd827fc700 1 -- 172.21.15.88:0/3583496929 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8410ba40 con 0x7fcd84106f20 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.080+0000 7fcd827fc700 1 -- 172.21.15.88:0/3583496929 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcd74001580 con 0x7fcd84106f20 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.080+0000 7fcd827fc700 1 -- 172.21.15.88:0/3583496929 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fcd74004930 con 0x7fcd84106f20 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.080+0000 7fcd827fc700 1 -- 172.21.15.88:0/3583496929 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcd74005a70 con 0x7fcd84106f20 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 -- 172.21.15.88:0/3583496929 >> v1:172.21.15.88:6789/0 conn(0x7fcd84106f20 legacy=0x7fcd84109310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 -- 172.21.15.88:0/3583496929 shutdown_connections 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 -- 172.21.15.88:0/3583496929 >> 172.21.15.88:0/3583496929 conn(0x7fcd841008f0 msgr2=0x7fcd84102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 -- 172.21.15.88:0/3583496929 shutdown_connections 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 -- 172.21.15.88:0/3583496929 wait complete. 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 Processor -- start 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 -- start start 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.081+0000 7fcd8993b700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd8419a7d0 con 0x7fcd84106f20 2024-03-26T21:16:40.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd82ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcd84106f20 0x7fcd8419a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54990/0 (socket says 172.21.15.88:54990) 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd82ffd700 1 -- 172.21.15.88:0/762659805 learned_addr learned my addr 172.21.15.88:0/762659805 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1421452184 0 0) 0x7fcd8419a7d0 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd70005530 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 904691286 0 0) 0x7fcd70005530 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd8419a7d0 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcd74001080 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1049983568 0 0) 0x7fcd8419a7d0 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8419a9a0 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8419ac90 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd8419e7e0 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fcd74004930 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.082+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcd74006050 con 0x7fcd84106f20 2024-03-26T21:16:40.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.083+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 8) ==== 46431+0+0 (unknown 2274444239 0 0) 0x7fcd74011760 con 0x7fcd84106f20 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.083+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd64005120 con 0x7fcd84106f20 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.083+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7fcd74047d10 con 0x7fcd84106f20 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.088+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fcd74017600 con 0x7fcd84106f20 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:39.358+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fcd64005410 con 0x7fcd84106f20 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.173+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (unknown 1498667528 0 0) 0x7fcd74016f00 con 0x7fcd84106f20 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.175+0000 7fcd88939700 1 -- 172.21.15.88:0/762659805 <== mon.0 v1:172.21.15.88:6789/0 11 ==== mgrmap(e 9) ==== 46444+0+0 (unknown 2165735970 0 0) 0x7fcd74046650 con 0x7fcd84106f20 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.177+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 >> v1:172.21.15.88:6800/3808082170 conn(0x7fcd7003a840 legacy=0x7fcd7003cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.177+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 >> v1:172.21.15.88:6789/0 conn(0x7fcd84106f20 legacy=0x7fcd8419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.177+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 shutdown_connections 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.177+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 >> 172.21.15.88:0/762659805 conn(0x7fcd841008f0 msgr2=0x7fcd84190d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.177+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 shutdown_connections 2024-03-26T21:16:40.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:40.177+0000 7fcd8993b700 1 -- 172.21.15.88:0/762659805 wait complete. 2024-03-26T21:16:41.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/762659805' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T21:16:41.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:41 smithi088 ceph-mon[30075]: mgrmap e9: y(active, since 20s) 2024-03-26T21:16:41.835 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:41 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:41.509+0000 7f7d65f36200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:16:41.835 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:41 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:41.602+0000 7f7d65f36200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:16:41.835 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:41 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:41.748+0000 7f7d65f36200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:16:41.835 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:41 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:41.841+0000 7f7d65f36200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:16:42.172 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:41 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:41.925+0000 7f7d65f36200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:16:42.192 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:16:42.192 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.314+0000 7fa68df9e700 1 Processor -- start 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.314+0000 7fa68df9e700 1 -- start start 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.314+0000 7fa68df9e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa688079e40 con 0x7fa688104d40 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.314+0000 7fa6877fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa688104d40 0x7fa688105160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55020/0 (socket says 172.21.15.88:55020) 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.314+0000 7fa6877fe700 1 -- 172.21.15.88:0/31653615 learned_addr learned my addr 172.21.15.88:0/31653615 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.315+0000 7fa6867fc700 1 -- 172.21.15.88:0/31653615 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1594604847 0 0) 0x7fa688079e40 con 0x7fa688104d40 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.315+0000 7fa6867fc700 1 -- 172.21.15.88:0/31653615 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6700035b0 con 0x7fa688104d40 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.315+0000 7fa6867fc700 1 -- 172.21.15.88:0/31653615 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1584960336 0 0) 0x7fa6700035b0 con 0x7fa688104d40 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.315+0000 7fa6867fc700 1 -- 172.21.15.88:0/31653615 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa688106880 con 0x7fa688104d40 2024-03-26T21:16:42.193 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.315+0000 7fa6867fc700 1 -- 172.21.15.88:0/31653615 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fa67c0018c0 con 0x7fa688104d40 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.315+0000 7fa6867fc700 1 -- 172.21.15.88:0/31653615 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fa67c001b20 con 0x7fa688104d40 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.315+0000 7fa6867fc700 1 -- 172.21.15.88:0/31653615 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fa67c005ab0 con 0x7fa688104d40 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.316+0000 7fa68df9e700 1 -- 172.21.15.88:0/31653615 >> v1:172.21.15.88:6789/0 conn(0x7fa688104d40 legacy=0x7fa688105160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.316+0000 7fa68df9e700 1 -- 172.21.15.88:0/31653615 shutdown_connections 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.316+0000 7fa68df9e700 1 -- 172.21.15.88:0/31653615 >> 172.21.15.88:0/31653615 conn(0x7fa6881008c0 msgr2=0x7fa688102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.316+0000 7fa68df9e700 1 -- 172.21.15.88:0/31653615 shutdown_connections 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.316+0000 7fa68df9e700 1 -- 172.21.15.88:0/31653615 wait complete. 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.316+0000 7fa68df9e700 1 Processor -- start 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa68df9e700 1 -- start start 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa68df9e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa68819c9c0 con 0x7fa688104d40 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa6877fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa688104d40 0x7fa68819c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55034/0 (socket says 172.21.15.88:55034) 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa6877fe700 1 -- 172.21.15.88:0/2443499990 learned_addr learned my addr 172.21.15.88:0/2443499990 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419424791 0 0) 0x7fa68819c9c0 con 0x7fa688104d40 2024-03-26T21:16:42.194 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa670005580 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2240223170 0 0) 0x7fa670005580 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa68819c9c0 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fa67c0013c0 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.317+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4248158341 0 0) 0x7fa68819c9c0 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.318+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68819cb90 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.318+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa68819ce80 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.318+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6881a09d0 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.318+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fa67c001b20 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.318+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fa67c0061c0 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.319+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 9) ==== 46444+0+0 (unknown 2165735970 0 0) 0x7fa67c0119a0 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.319+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa674005150 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.319+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7fa67c048000 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.319+0000 7fa686ffd700 1 -- 172.21.15.88:0/2443499990 >> v1:172.21.15.88:6800/3808082170 conn(0x7fa67003a880 legacy=0x7fa67003cd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/3808082170 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.324+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fa67c017930 con 0x7fa688104d40 2024-03-26T21:16:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.520+0000 7fa686ffd700 1 -- 172.21.15.88:0/2443499990 >> v1:172.21.15.88:6800/3808082170 conn(0x7fa67003a880 legacy=0x7fa67003cd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/3808082170 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.609+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa674005c50 con 0x7fa688104d40 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.610+0000 7fa684ff9700 1 -- 172.21.15.88:0/2443499990 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7fa67c017230 con 0x7fa688104d40 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.612+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 >> v1:172.21.15.88:6800/3808082170 conn(0x7fa67003a880 legacy=0x7fa67003cd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.612+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 >> v1:172.21.15.88:6789/0 conn(0x7fa688104d40 legacy=0x7fa68819c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.612+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 shutdown_connections 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.612+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 >> 172.21.15.88:0/2443499990 conn(0x7fa6881008c0 msgr2=0x7fa688102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.612+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 shutdown_connections 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:41.612+0000 7fa68df9e700 1 -- 172.21.15.88:0/2443499990 wait complete. 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for the mgr to restart... 2024-03-26T21:16:42.196 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr epoch 9... 2024-03-26T21:16:42.434 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2443499990' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T21:16:42.435 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:42.189+0000 7f7d65f36200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:16:42.435 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:42.309+0000 7f7d65f36200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:16:42.435 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:42.394+0000 7f7d65f36200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:16:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:42.494+0000 7f7d65f36200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:16:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:42.673+0000 7f7d65f36200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:16:43.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:43 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:43.627+0000 7f7d65f36200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:16:43.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:43 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:43.757+0000 7f7d65f36200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:16:44.365 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:44.084+0000 7f7d65f36200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:16:44.365 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:44.174+0000 7f7d65f36200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:16:44.365 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:44.371+0000 7f7d65f36200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:16:44.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:44.554+0000 7f7d65f36200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:16:45.089 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:44.839+0000 7f7d65f36200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:16:45.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:45 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:45.095+0000 7f7d65f36200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:16:45.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:45 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:45.334+0000 7f7d65f36200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:16:46.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:46 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:46.141+0000 7f7d65f36200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:16:46.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:46 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:46.734+0000 7f7d65f36200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:16:46.984 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:46 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:46.832+0000 7f7d65f36200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:16:47.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:16:47.050+0000 7f7d65f36200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:16:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: Active manager daemon y restarted 2024-03-26T21:16:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: Activating manager daemon y 2024-03-26T21:16:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: mgrmap e10: y(active, starting, since 0.00322344s) 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: Manager daemon y is now available 2024-03-26T21:16:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:48.367 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:16:48.368 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:16:48.368 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:48 smithi088 ceph-mon[30075]: mgrmap e11: y(active, since 1.00524s) 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.681+0000 7ffa6faaf700 1 Processor -- start 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.681+0000 7ffa6faaf700 1 -- start start 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.681+0000 7ffa6faaf700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffa68079e80 con 0x7ffa68106f10 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.681+0000 7ffa6d84b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffa68106f10 0x7ffa68109300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55038/0 (socket says 172.21.15.88:55038) 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.681+0000 7ffa6d84b700 1 -- 172.21.15.88:0/2408194694 learned_addr learned my addr 172.21.15.88:0/2408194694 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:48.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.682+0000 7ffa6c849700 1 -- 172.21.15.88:0/2408194694 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3042890131 0 0) 0x7ffa68079e80 con 0x7ffa68106f10 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.682+0000 7ffa6c849700 1 -- 172.21.15.88:0/2408194694 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa540035b0 con 0x7ffa68106f10 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.682+0000 7ffa6c849700 1 -- 172.21.15.88:0/2408194694 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1882619626 0 0) 0x7ffa540035b0 con 0x7ffa68106f10 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.682+0000 7ffa6c849700 1 -- 172.21.15.88:0/2408194694 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6810ba30 con 0x7ffa68106f10 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.682+0000 7ffa6c849700 1 -- 172.21.15.88:0/2408194694 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ffa640018c0 con 0x7ffa68106f10 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.682+0000 7ffa6c849700 1 -- 172.21.15.88:0/2408194694 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7ffa64001bf0 con 0x7ffa68106f10 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.682+0000 7ffa6c849700 1 -- 172.21.15.88:0/2408194694 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ffa64005b70 con 0x7ffa68106f10 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 -- 172.21.15.88:0/2408194694 >> v1:172.21.15.88:6789/0 conn(0x7ffa68106f10 legacy=0x7ffa68109300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 -- 172.21.15.88:0/2408194694 shutdown_connections 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 -- 172.21.15.88:0/2408194694 >> 172.21.15.88:0/2408194694 conn(0x7ffa681008c0 msgr2=0x7ffa68102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 -- 172.21.15.88:0/2408194694 shutdown_connections 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 -- 172.21.15.88:0/2408194694 wait complete. 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 Processor -- start 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 -- start start 2024-03-26T21:16:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.683+0000 7ffa6faaf700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffa6819a780 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa6d84b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffa68106f10 0x7ffa6819a070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55044/0 (socket says 172.21.15.88:55044) 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa6d84b700 1 -- 172.21.15.88:0/813901030 learned_addr learned my addr 172.21.15.88:0/813901030 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1316400859 0 0) 0x7ffa6819a780 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa54005530 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2405060938 0 0) 0x7ffa54005530 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa6819a780 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ffa64001130 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2858620742 0 0) 0x7ffa6819a780 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6819a950 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6819ac40 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffa6819e790 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7ffa64001bf0 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.684+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ffa640061c0 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.685+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 9) ==== 46444+0+0 (unknown 2165735970 0 0) 0x7ffa64011900 con 0x7ffa68106f10 2024-03-26T21:16:48.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.685+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6800/3808082170 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffa5403e410 con 0x7ffa5403a740 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.686+0000 7ffa6d04a700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6800/3808082170 conn(0x7ffa5403a740 legacy=0x7ffa5403cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/3808082170 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.686+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3276581244 0 0) 0x7ffa64047fe0 con 0x7ffa68106f10 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:42.887+0000 7ffa6d04a700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6800/3808082170 conn(0x7ffa5403a740 legacy=0x7ffa5403cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/3808082170 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:43.288+0000 7ffa6d04a700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6800/3808082170 conn(0x7ffa5403a740 legacy=0x7ffa5403cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/3808082170 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:44.089+0000 7ffa6d04a700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6800/3808082170 conn(0x7ffa5403a740 legacy=0x7ffa5403cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/3808082170 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:45.690+0000 7ffa6d04a700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6800/3808082170 conn(0x7ffa5403a740 legacy=0x7ffa5403cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/3808082170 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:47.066+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mgrmap(e 10) ==== 46246+0+0 (unknown 2659064919 0 0) 0x7ffa64012f10 con 0x7ffa68106f10 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:47.066+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6800/3808082170 conn(0x7ffa5403a740 legacy=0x7ffa5403cc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:16:48.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.067+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mgrmap(e 11) ==== 46338+0+0 (unknown 3987204605 0 0) 0x7ffa64049320 con 0x7ffa68106f10 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.067+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6800/4097082509 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffa5403e410 con 0x7ffa5403efd0 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.068+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7ffa5403e410 con 0x7ffa5403efd0 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.075+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 --> v1:172.21.15.88:6800/4097082509 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ffa4c002630 con 0x7ffa5403efd0 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.075+0000 7ffa5effd700 1 -- 172.21.15.88:0/813901030 <== mgr.14150 v1:172.21.15.88:6800/4097082509 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (unknown 0 0 3086460295) 0x7ffa4c002630 con 0x7ffa5403efd0 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.076+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6800/4097082509 conn(0x7ffa5403efd0 legacy=0x7ffa540413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.076+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 >> v1:172.21.15.88:6789/0 conn(0x7ffa68106f10 legacy=0x7ffa6819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.076+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 shutdown_connections 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.076+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 >> 172.21.15.88:0/813901030 conn(0x7ffa681008c0 msgr2=0x7ffa68190d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.076+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 shutdown_connections 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:48.076+0000 7ffa6faaf700 1 -- 172.21.15.88:0/813901030 wait complete. 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:mgr epoch 9 is available 2024-03-26T21:16:48.687 INFO:teuthology.orchestra.run.smithi088.stdout:Generating a dashboard self-signed certificate... 2024-03-26T21:16:49.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:49 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:47] ENGINE Bus STARTING 2024-03-26T21:16:49.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:49 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:48] ENGINE Serving on http://172.21.15.88:8765 2024-03-26T21:16:49.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:49 smithi088 ceph-mon[30075]: from='client.14154 v1:172.21.15.88:0/813901030' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T21:16:49.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:49 smithi088 ceph-mon[30075]: from='client.14154 v1:172.21.15.88:0/813901030' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T21:16:49.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:49 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:48] ENGINE Serving on https://172.21.15.88:7150 2024-03-26T21:16:49.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:49 smithi088 ceph-mon[30075]: [26/Mar/2024:21:16:48] ENGINE Bus STARTED 2024-03-26T21:16:49.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:50.435 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.512+0000 7f0e30695700 1 Processor -- start 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.512+0000 7f0e30695700 1 -- start start 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e30695700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e28109780 con 0x7f0e28105350 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e2e431700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0e28105350 0x7f0e28105770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51556/0 (socket says 172.21.15.88:51556) 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e2e431700 1 -- 172.21.15.88:0/1676387290 learned_addr learned my addr 172.21.15.88:0/1676387290 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e2d42f700 1 -- 172.21.15.88:0/1676387290 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1098642152 0 0) 0x7f0e28109780 con 0x7f0e28105350 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e2d42f700 1 -- 172.21.15.88:0/1676387290 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e140035b0 con 0x7f0e28105350 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e2d42f700 1 -- 172.21.15.88:0/1676387290 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 785581064 0 0) 0x7f0e140035b0 con 0x7f0e28105350 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e2d42f700 1 -- 172.21.15.88:0/1676387290 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2810a960 con 0x7f0e28105350 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.513+0000 7f0e2d42f700 1 -- 172.21.15.88:0/1676387290 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f0e24001580 con 0x7f0e28105350 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.514+0000 7f0e2d42f700 1 -- 172.21.15.88:0/1676387290 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f0e24004930 con 0x7f0e28105350 2024-03-26T21:16:50.436 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.514+0000 7f0e2d42f700 1 -- 172.21.15.88:0/1676387290 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f0e24005960 con 0x7f0e28105350 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.514+0000 7f0e30695700 1 -- 172.21.15.88:0/1676387290 >> v1:172.21.15.88:6789/0 conn(0x7f0e28105350 legacy=0x7f0e28105770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.514+0000 7f0e30695700 1 -- 172.21.15.88:0/1676387290 shutdown_connections 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.514+0000 7f0e30695700 1 -- 172.21.15.88:0/1676387290 >> 172.21.15.88:0/1676387290 conn(0x7f0e281008d0 msgr2=0x7f0e28102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.514+0000 7f0e30695700 1 -- 172.21.15.88:0/1676387290 shutdown_connections 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.514+0000 7f0e30695700 1 -- 172.21.15.88:0/1676387290 wait complete. 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.515+0000 7f0e30695700 1 Processor -- start 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.515+0000 7f0e30695700 1 -- start start 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.515+0000 7f0e30695700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e2807a310 con 0x7f0e28105350 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.515+0000 7f0e2e431700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0e28105350 0x7f0e2819cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51558/0 (socket says 172.21.15.88:51558) 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.515+0000 7f0e2e431700 1 -- 172.21.15.88:0/3925719040 learned_addr learned my addr 172.21.15.88:0/3925719040 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830382072 0 0) 0x7f0e2807a310 con 0x7f0e28105350 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e14005530 con 0x7f0e28105350 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2724392094 0 0) 0x7f0e14005530 con 0x7f0e28105350 2024-03-26T21:16:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e2807a310 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f0e24001080 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 994253592 0 0) 0x7f0e2807a310 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2807a4e0 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2819e4b0 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e2819e850 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f0e24004930 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.516+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f0e24005f90 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.517+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e0c005150 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.517+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 11) ==== 46338+0+0 (unknown 3987204605 0 0) 0x7f0e24011640 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.517+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7f0e24047b20 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.522+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f0e240175a0 con 0x7f0e28105350 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.742+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f0e0c000bc0 con 0x7f0e1403a7c0 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.860+0000 7f0e1f7fe700 1 -- 172.21.15.88:0/3925719040 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (unknown 0 0 3317752739) 0x7f0e0c000bc0 con 0x7f0e1403a7c0 2024-03-26T21:16:50.438 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.862+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 >> v1:172.21.15.88:6800/4097082509 conn(0x7f0e1403a7c0 legacy=0x7f0e1403cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:50.439 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.862+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 >> v1:172.21.15.88:6789/0 conn(0x7f0e28105350 legacy=0x7f0e2819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:50.439 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.862+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 shutdown_connections 2024-03-26T21:16:50.439 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.862+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 >> 172.21.15.88:0/3925719040 conn(0x7f0e281008d0 msgr2=0x7f0e28102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:50.439 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.862+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 shutdown_connections 2024-03-26T21:16:50.439 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:49.862+0000 7f0e30695700 1 -- 172.21.15.88:0/3925719040 wait complete. 2024-03-26T21:16:50.439 INFO:teuthology.orchestra.run.smithi088.stdout:Creating initial admin user... 2024-03-26T21:16:51.107 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:50 smithi088 ceph-mon[30075]: from='client.14162 v1:172.21.15.88:0/3925719040' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:51.107 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:51.107 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:51.107 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:50 smithi088 ceph-mon[30075]: mgrmap e12: y(active, since 2s) 2024-03-26T21:16:51.107 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:51.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:51 smithi088 ceph-mon[30075]: from='client.14164 v1:172.21.15.88:0/1224749071' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:16:51.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:52.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$JTL/c4P2.AmSw8ozeivnoOu0O5VEI..lsW69IuR8tlgj.KGkrEWWW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711487811, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T21:16:52.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.884+0000 7f91103a7700 1 Processor -- start 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.884+0000 7f91103a7700 1 -- start start 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.884+0000 7f91103a7700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f910810a910 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.884+0000 7f910e143700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9108104750 0x7f9108104b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51574/0 (socket says 172.21.15.88:51574) 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.884+0000 7f910e143700 1 -- 172.21.15.88:0/1835770898 learned_addr learned my addr 172.21.15.88:0/1835770898 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.885+0000 7f910d141700 1 -- 172.21.15.88:0/1835770898 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1952217565 0 0) 0x7f910810a910 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.885+0000 7f910d141700 1 -- 172.21.15.88:0/1835770898 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90f40035b0 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.885+0000 7f910d141700 1 -- 172.21.15.88:0/1835770898 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1106377468 0 0) 0x7f90f40035b0 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.885+0000 7f910d141700 1 -- 172.21.15.88:0/1835770898 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910810baf0 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.885+0000 7f910d141700 1 -- 172.21.15.88:0/1835770898 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f91040018c0 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.885+0000 7f910d141700 1 -- 172.21.15.88:0/1835770898 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f9104001bf0 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.885+0000 7f910d141700 1 -- 172.21.15.88:0/1835770898 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f9104005b70 con 0x7f9108104750 2024-03-26T21:16:52.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.886+0000 7f91103a7700 1 -- 172.21.15.88:0/1835770898 >> v1:172.21.15.88:6789/0 conn(0x7f9108104750 legacy=0x7f9108104b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.886+0000 7f91103a7700 1 -- 172.21.15.88:0/1835770898 shutdown_connections 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.886+0000 7f91103a7700 1 -- 172.21.15.88:0/1835770898 >> 172.21.15.88:0/1835770898 conn(0x7f9108100040 msgr2=0x7f91081024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.886+0000 7f91103a7700 1 -- 172.21.15.88:0/1835770898 shutdown_connections 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.886+0000 7f91103a7700 1 -- 172.21.15.88:0/1835770898 wait complete. 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.886+0000 7f91103a7700 1 Processor -- start 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f91103a7700 1 -- start start 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f91103a7700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f910819a780 con 0x7f9108104750 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f910e143700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9108104750 0x7f910819a070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51590/0 (socket says 172.21.15.88:51590) 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f910e143700 1 -- 172.21.15.88:0/1224749071 learned_addr learned my addr 172.21.15.88:0/1224749071 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3416185320 0 0) 0x7f910819a780 con 0x7f9108104750 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90f4005530 con 0x7f9108104750 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1576003071 0 0) 0x7f90f4005530 con 0x7f9108104750 2024-03-26T21:16:52.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f910819a780 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f9104001130 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2898279456 0 0) 0x7f910819a780 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910819a950 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.887+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f910819ac40 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.888+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f910819e790 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.888+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f9104001500 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.888+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f9104006440 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.889+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 12) ==== 46444+0+0 (unknown 3244711609 0 0) 0x7f9104011b80 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.889+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90ec005150 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.889+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7f9104048240 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:50.894+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f9104017b00 con 0x7f9108104750 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.112+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f90ec002430 con 0x7f90f403a6a0 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.338+0000 7f90ff7fe700 1 -- 172.21.15.88:0/1224749071 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (unknown 0 0 644544703) 0x7f90ec002430 con 0x7f90f403a6a0 2024-03-26T21:16:52.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.340+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 >> v1:172.21.15.88:6800/4097082509 conn(0x7f90f403a6a0 legacy=0x7f90f403cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:52.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.340+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 >> v1:172.21.15.88:6789/0 conn(0x7f9108104750 legacy=0x7f910819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:52.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.340+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 shutdown_connections 2024-03-26T21:16:52.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.340+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 >> 172.21.15.88:0/1224749071 conn(0x7f9108100040 msgr2=0x7f91081024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:52.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.341+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 shutdown_connections 2024-03-26T21:16:52.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:51.341+0000 7f91103a7700 1 -- 172.21.15.88:0/1224749071 wait complete. 2024-03-26T21:16:52.420 INFO:teuthology.orchestra.run.smithi088.stdout:Fetching dashboard port number... 2024-03-26T21:16:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2484173410' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T21:16:53.758 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T21:16:53.758 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.845+0000 7fcc99807700 1 Processor -- start 2024-03-26T21:16:53.758 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.845+0000 7fcc99807700 1 -- start start 2024-03-26T21:16:53.758 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.845+0000 7fcc99807700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc9407a6f0 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc92ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcc94106f70 0x7fcc94109360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51604/0 (socket says 172.21.15.88:51604) 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc92ffd700 1 -- 172.21.15.88:0/1212199406 learned_addr learned my addr 172.21.15.88:0/1212199406 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc91ffb700 1 -- 172.21.15.88:0/1212199406 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602265207 0 0) 0x7fcc9407a6f0 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc91ffb700 1 -- 172.21.15.88:0/1212199406 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc7c0035b0 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc91ffb700 1 -- 172.21.15.88:0/1212199406 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3201313952 0 0) 0x7fcc7c0035b0 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc91ffb700 1 -- 172.21.15.88:0/1212199406 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9410ba90 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc91ffb700 1 -- 172.21.15.88:0/1212199406 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcc88001580 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc91ffb700 1 -- 172.21.15.88:0/1212199406 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fcc88004930 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.846+0000 7fcc91ffb700 1 -- 172.21.15.88:0/1212199406 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcc88005a70 con 0x7fcc94106f70 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.847+0000 7fcc99807700 1 -- 172.21.15.88:0/1212199406 >> v1:172.21.15.88:6789/0 conn(0x7fcc94106f70 legacy=0x7fcc94109360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.847+0000 7fcc99807700 1 -- 172.21.15.88:0/1212199406 shutdown_connections 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.847+0000 7fcc99807700 1 -- 172.21.15.88:0/1212199406 >> 172.21.15.88:0/1212199406 conn(0x7fcc94100920 msgr2=0x7fcc94102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:53.759 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.847+0000 7fcc99807700 1 -- 172.21.15.88:0/1212199406 shutdown_connections 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.847+0000 7fcc99807700 1 -- 172.21.15.88:0/1212199406 wait complete. 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.847+0000 7fcc99807700 1 Processor -- start 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.847+0000 7fcc99807700 1 -- start start 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc99807700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc9419a740 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc92ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcc94106f70 0x7fcc9419a030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51614/0 (socket says 172.21.15.88:51614) 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc92ffd700 1 -- 172.21.15.88:0/2484173410 learned_addr learned my addr 172.21.15.88:0/2484173410 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237327359 0 0) 0x7fcc9419a740 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc7c005530 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 201729444 0 0) 0x7fcc7c005530 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc9419a740 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcc88001080 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2021114306 0 0) 0x7fcc9419a740 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9419a910 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9419ac00 con 0x7fcc94106f70 2024-03-26T21:16:53.760 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc9419e750 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fcc88004930 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.848+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fcc880060e0 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.849+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 12) ==== 46444+0+0 (unknown 3244711609 0 0) 0x7fcc88011820 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.849+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc80005150 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.850+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7fcc88047f20 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:52.854+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fcc880177a0 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.058+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fcc80005440 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.059+0000 7fcc98805700 1 -- 172.21.15.88:0/2484173410 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7fcc880170a0 con 0x7fcc94106f70 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.061+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 >> v1:172.21.15.88:6800/4097082509 conn(0x7fcc7c03a840 legacy=0x7fcc7c03cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.061+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 >> v1:172.21.15.88:6789/0 conn(0x7fcc94106f70 legacy=0x7fcc9419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.061+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 shutdown_connections 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.061+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 >> 172.21.15.88:0/2484173410 conn(0x7fcc94100920 msgr2=0x7fcc94190d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.061+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 shutdown_connections 2024-03-26T21:16:53.761 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:53.061+0000 7fcc99807700 1 -- 172.21.15.88:0/2484173410 wait complete. 2024-03-26T21:16:53.768 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:16:53.768 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2024-03-26T21:16:53.775 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:16:53.775 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2024-03-26T21:16:53.775 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2024-03-26T21:16:53.775 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T21:16:53.776 INFO:teuthology.orchestra.run.smithi088.stdout:Ceph Dashboard is now available at: 2024-03-26T21:16:53.777 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:53.777 INFO:teuthology.orchestra.run.smithi088.stdout: URL: https://smithi088.front.sepia.ceph.com:8443/ 2024-03-26T21:16:53.777 INFO:teuthology.orchestra.run.smithi088.stdout: User: admin 2024-03-26T21:16:53.777 INFO:teuthology.orchestra.run.smithi088.stdout: Password: w18okvvcfl 2024-03-26T21:16:53.777 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:53.777 INFO:teuthology.orchestra.run.smithi088.stdout:Saving cluster configuration to /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config directory 2024-03-26T21:16:53.778 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling autotune for osd_memory_target 2024-03-26T21:16:54.460 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:54 smithi088 ceph-mon[30075]: mgrmap e13: y(active, since 6s) 2024-03-26T21:16:55.023 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.262+0000 7f79b2957700 1 Processor -- start 2024-03-26T21:16:55.023 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.262+0000 7f79b2957700 1 -- start start 2024-03-26T21:16:55.023 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.262+0000 7f79b2957700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79ac10a8c0 con 0x7f79ac106490 2024-03-26T21:16:55.023 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.262+0000 7f79abfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f79ac106490 0x7f79ac1068b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45526/0 (socket says 172.21.15.88:45526) 2024-03-26T21:16:55.023 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.262+0000 7f79abfff700 1 -- 172.21.15.88:0/811479645 learned_addr learned my addr 172.21.15.88:0/811479645 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.263+0000 7f79aaffd700 1 -- 172.21.15.88:0/811479645 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915402757 0 0) 0x7f79ac10a8c0 con 0x7f79ac106490 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.263+0000 7f79aaffd700 1 -- 172.21.15.88:0/811479645 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79940035b0 con 0x7f79ac106490 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.263+0000 7f79aaffd700 1 -- 172.21.15.88:0/811479645 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3484252847 0 0) 0x7f79940035b0 con 0x7f79ac106490 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.263+0000 7f79aaffd700 1 -- 172.21.15.88:0/811479645 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac10baa0 con 0x7f79ac106490 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.263+0000 7f79aaffd700 1 -- 172.21.15.88:0/811479645 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f799c001580 con 0x7f79ac106490 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.263+0000 7f79aaffd700 1 -- 172.21.15.88:0/811479645 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f799c004930 con 0x7f79ac106490 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.263+0000 7f79aaffd700 1 -- 172.21.15.88:0/811479645 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f799c005960 con 0x7f79ac106490 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.264+0000 7f79b2957700 1 -- 172.21.15.88:0/811479645 >> v1:172.21.15.88:6789/0 conn(0x7f79ac106490 legacy=0x7f79ac1068b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.264+0000 7f79b2957700 1 -- 172.21.15.88:0/811479645 shutdown_connections 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.264+0000 7f79b2957700 1 -- 172.21.15.88:0/811479645 >> 172.21.15.88:0/811479645 conn(0x7f79ac101a30 msgr2=0x7f79ac103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.264+0000 7f79b2957700 1 -- 172.21.15.88:0/811479645 shutdown_connections 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.264+0000 7f79b2957700 1 -- 172.21.15.88:0/811479645 wait complete. 2024-03-26T21:16:55.024 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.264+0000 7f79b2957700 1 Processor -- start 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.264+0000 7f79b2957700 1 -- start start 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79b2957700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79ac19a740 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79abfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f79ac106490 0x7f79ac19a030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45534/0 (socket says 172.21.15.88:45534) 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79abfff700 1 -- 172.21.15.88:0/2225784508 learned_addr learned my addr 172.21.15.88:0/2225784508 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3204888574 0 0) 0x7f79ac19a740 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7994005530 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2282597194 0 0) 0x7f7994005530 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79ac19a740 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f799c001080 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3914209725 0 0) 0x7f79ac19a740 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac19a910 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79ac19ac00 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f79ac19e750 con 0x7f79ac106490 2024-03-26T21:16:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f799c004930 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.265+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f799c005fc0 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.266+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f799c0117c0 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.266+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7998005150 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.266+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7f799c047f50 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.271+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f799c0177e0 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.466+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 --> v1:172.21.15.88:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f7998005440 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.466+0000 7f79a97fa700 1 -- 172.21.15.88:0/2225784508 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (unknown 689008156 0 0) 0x7f799c0170e0 con 0x7f79ac106490 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.468+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 >> v1:172.21.15.88:6800/4097082509 conn(0x7f799403a8e0 legacy=0x7f799403cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.468+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 >> v1:172.21.15.88:6789/0 conn(0x7f79ac106490 legacy=0x7f79ac19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.468+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 shutdown_connections 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.468+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 >> 172.21.15.88:0/2225784508 conn(0x7f79ac101a30 msgr2=0x7f79ac103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.468+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 shutdown_connections 2024-03-26T21:16:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:54.468+0000 7f79b2957700 1 -- 172.21.15.88:0/2225784508 wait complete. 2024-03-26T21:16:56.345 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.458+0000 7f18b95c7700 1 Processor -- start 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.458+0000 7f18b95c7700 1 -- start start 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.458+0000 7f18b95c7700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18b407a6d0 con 0x7f18b4104d80 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.458+0000 7f18b2ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f18b4104d80 0x7f18b41051a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45538/0 (socket says 172.21.15.88:45538) 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.458+0000 7f18b2ffd700 1 -- 172.21.15.88:0/3776526272 learned_addr learned my addr 172.21.15.88:0/3776526272 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.459+0000 7f18b27fc700 1 -- 172.21.15.88:0/3776526272 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862407010 0 0) 0x7f18b407a6d0 con 0x7f18b4104d80 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.459+0000 7f18b27fc700 1 -- 172.21.15.88:0/3776526272 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a00035b0 con 0x7f18b4104d80 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.459+0000 7f18b27fc700 1 -- 172.21.15.88:0/3776526272 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4003119519 0 0) 0x7f18a00035b0 con 0x7f18b4104d80 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.459+0000 7f18b27fc700 1 -- 172.21.15.88:0/3776526272 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b41068c0 con 0x7f18b4104d80 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.459+0000 7f18b27fc700 1 -- 172.21.15.88:0/3776526272 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f18a4001580 con 0x7f18b4104d80 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.459+0000 7f18b27fc700 1 -- 172.21.15.88:0/3776526272 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f18a4004930 con 0x7f18b4104d80 2024-03-26T21:16:56.346 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.459+0000 7f18b27fc700 1 -- 172.21.15.88:0/3776526272 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f18a4005a70 con 0x7f18b4104d80 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.460+0000 7f18b95c7700 1 -- 172.21.15.88:0/3776526272 >> v1:172.21.15.88:6789/0 conn(0x7f18b4104d80 legacy=0x7f18b41051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.460+0000 7f18b95c7700 1 -- 172.21.15.88:0/3776526272 shutdown_connections 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.460+0000 7f18b95c7700 1 -- 172.21.15.88:0/3776526272 >> 172.21.15.88:0/3776526272 conn(0x7f18b4100920 msgr2=0x7f18b4102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.460+0000 7f18b95c7700 1 -- 172.21.15.88:0/3776526272 shutdown_connections 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.460+0000 7f18b95c7700 1 -- 172.21.15.88:0/3776526272 wait complete. 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.460+0000 7f18b95c7700 1 Processor -- start 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.460+0000 7f18b95c7700 1 -- start start 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b95c7700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18b419c930 con 0x7f18b4104d80 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b2ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f18b4104d80 0x7f18b419c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45542/0 (socket says 172.21.15.88:45542) 2024-03-26T21:16:56.347 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b2ffd700 1 -- 172.21.15.88:0/3243014717 learned_addr learned my addr 172.21.15.88:0/3243014717 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2686917599 0 0) 0x7f18b419c930 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a0005530 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2485004494 0 0) 0x7f18a0005530 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18b419c930 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f18a4001080 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 703985063 0 0) 0x7f18b419c930 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b419cb00 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18b419cdf0 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.461+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f18b41a0940 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.462+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f18a4004930 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.462+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f18a40060e0 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.462+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f18a4011840 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.462+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1894005150 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.462+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7f18a4048e40 con 0x7f18b4104d80 2024-03-26T21:16:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.467+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f18a4017860 con 0x7f18b4104d80 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.777+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 --> v1:172.21.15.88:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1894005440 con 0x7f18b4104d80 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.779+0000 7f18b0ff9700 1 -- 172.21.15.88:0/3243014717 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (unknown 38603050 0 0) 0x7f18a4017160 con 0x7f18b4104d80 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.782+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 >> v1:172.21.15.88:6800/4097082509 conn(0x7f18a003a8e0 legacy=0x7f18a003cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.782+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 >> v1:172.21.15.88:6789/0 conn(0x7f18b4104d80 legacy=0x7f18b419c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.782+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 shutdown_connections 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.782+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 >> 172.21.15.88:0/3243014717 conn(0x7f18b4100920 msgr2=0x7f18b4102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.782+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 shutdown_connections 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2024-03-26T21:16:55.782+0000 7f18b95c7700 1 -- 172.21.15.88:0/3243014717 wait complete. 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout: sudo /sbin/cephadm shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T21:16:56.349 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: sudo /sbin/cephadm shell 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph telemetry on 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout:For more information see: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:16:56.350 INFO:teuthology.orchestra.run.smithi088.stdout:Bootstrap complete. 2024-03-26T21:16:56.385 INFO:tasks.cephadm:Fetching config... 2024-03-26T21:16:56.386 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:16:56.386 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T21:16:56.401 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T21:16:56.401 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:16:56.401 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T21:16:56.455 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T21:16:56.455 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:16:56.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T21:16:56.523 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T21:16:56.523 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:16:56.523 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T21:16:56.580 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T21:16:56.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUWO/Y48EDmfHaNiYlajyX6tELb0BWkW4JOsQX+RybShngd24Gy3oj4qLtRe5bHhVWkN0CRZqS0SSoAzCeTzgsr15pQfBzWWRaXDOaDzbBPqj/6lQamVv+/V1pIXaHdBUe3r9BwWFbbe+Dg2Zw1flB3szBMatBHWh//zWPDe1clCCwZm5yF2Bvdd6P7ySXyPOp/H8h+AkfmjIxzz/PKUg0mOsOchKevDTfYp2R/pS1RRMt8VB1IgHskqkwEbKKsGfe6knoMDPAwr0a7phok+M+Mh6Uu5XG53PqtMOePyeNFmT6SXBDBBcDHCZVz6F7JveUlI+XnEMWSAP8MCNaiC+FHZ+ZdcQTWTIpPJiV1clv8f2CFBEThuB7tSXcEiRrD1xhgGri6l16AsQ0cZixcW1Sm76Z+NRR1Kkr67ScDfRvooPsw5Dd8jPnFMVnBNz9IM92A2pRKeFMGgR5jf4WN1YjJeX7BnadJ4Zbuq18P3IgpYabkTyAHftfOqt3B2X5lRs= ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T21:16:56.663 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUWO/Y48EDmfHaNiYlajyX6tELb0BWkW4JOsQX+RybShngd24Gy3oj4qLtRe5bHhVWkN0CRZqS0SSoAzCeTzgsr15pQfBzWWRaXDOaDzbBPqj/6lQamVv+/V1pIXaHdBUe3r9BwWFbbe+Dg2Zw1flB3szBMatBHWh//zWPDe1clCCwZm5yF2Bvdd6P7ySXyPOp/H8h+AkfmjIxzz/PKUg0mOsOchKevDTfYp2R/pS1RRMt8VB1IgHskqkwEbKKsGfe6knoMDPAwr0a7phok+M+Mh6Uu5XG53PqtMOePyeNFmT6SXBDBBcDHCZVz6F7JveUlI+XnEMWSAP8MCNaiC+FHZ+ZdcQTWTIpPJiV1clv8f2CFBEThuB7tSXcEiRrD1xhgGri6l16AsQ0cZixcW1Sm76Z+NRR1Kkr67ScDfRvooPsw5Dd8jPnFMVnBNz9IM92A2pRKeFMGgR5jf4WN1YjJeX7BnadJ4Zbuq18P3IgpYabkTyAHftfOqt3B2X5lRs= ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:16:56.678 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUWO/Y48EDmfHaNiYlajyX6tELb0BWkW4JOsQX+RybShngd24Gy3oj4qLtRe5bHhVWkN0CRZqS0SSoAzCeTzgsr15pQfBzWWRaXDOaDzbBPqj/6lQamVv+/V1pIXaHdBUe3r9BwWFbbe+Dg2Zw1flB3szBMatBHWh//zWPDe1clCCwZm5yF2Bvdd6P7ySXyPOp/H8h+AkfmjIxzz/PKUg0mOsOchKevDTfYp2R/pS1RRMt8VB1IgHskqkwEbKKsGfe6knoMDPAwr0a7phok+M+Mh6Uu5XG53PqtMOePyeNFmT6SXBDBBcDHCZVz6F7JveUlI+XnEMWSAP8MCNaiC+FHZ+ZdcQTWTIpPJiV1clv8f2CFBEThuB7tSXcEiRrD1xhgGri6l16AsQ0cZixcW1Sm76Z+NRR1Kkr67ScDfRvooPsw5Dd8jPnFMVnBNz9IM92A2pRKeFMGgR5jf4WN1YjJeX7BnadJ4Zbuq18P3IgpYabkTyAHftfOqt3B2X5lRs= ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T21:16:56.723 INFO:teuthology.orchestra.run.smithi138.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUWO/Y48EDmfHaNiYlajyX6tELb0BWkW4JOsQX+RybShngd24Gy3oj4qLtRe5bHhVWkN0CRZqS0SSoAzCeTzgsr15pQfBzWWRaXDOaDzbBPqj/6lQamVv+/V1pIXaHdBUe3r9BwWFbbe+Dg2Zw1flB3szBMatBHWh//zWPDe1clCCwZm5yF2Bvdd6P7ySXyPOp/H8h+AkfmjIxzz/PKUg0mOsOchKevDTfYp2R/pS1RRMt8VB1IgHskqkwEbKKsGfe6knoMDPAwr0a7phok+M+Mh6Uu5XG53PqtMOePyeNFmT6SXBDBBcDHCZVz6F7JveUlI+XnEMWSAP8MCNaiC+FHZ+ZdcQTWTIpPJiV1clv8f2CFBEThuB7tSXcEiRrD1xhgGri6l16AsQ0cZixcW1Sm76Z+NRR1Kkr67ScDfRvooPsw5Dd8jPnFMVnBNz9IM92A2pRKeFMGgR5jf4WN1YjJeX7BnadJ4Zbuq18P3IgpYabkTyAHftfOqt3B2X5lRs= ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:16:56.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T21:16:56.978 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:16:56.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3243014717' entity='client.admin' 2024-03-26T21:16:59.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.223+0000 7f8685dfd700 1 -- 172.21.15.88:0/3436259506 >> v1:172.21.15.88:6789/0 conn(0x7f8680100650 legacy=0x7f8680100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:59.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.223+0000 7f8685dfd700 1 -- 172.21.15.88:0/3436259506 shutdown_connections 2024-03-26T21:16:59.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.223+0000 7f8685dfd700 1 -- 172.21.15.88:0/3436259506 >> 172.21.15.88:0/3436259506 conn(0x7f86800fbc10 msgr2=0x7f86800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:59.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.223+0000 7f8685dfd700 1 -- 172.21.15.88:0/3436259506 shutdown_connections 2024-03-26T21:16:59.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.223+0000 7f8685dfd700 1 -- 172.21.15.88:0/3436259506 wait complete. 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.224+0000 7f8685dfd700 1 Processor -- start 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.224+0000 7f8685dfd700 1 -- start start 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.224+0000 7f8685dfd700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f86801970e0 con 0x7f8680100650 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.224+0000 7f867f7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8680100650 0x7f86801969d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45552/0 (socket says 172.21.15.88:45552) 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.224+0000 7f867f7fe700 1 -- 172.21.15.88:0/283539009 learned_addr learned my addr 172.21.15.88:0/283539009 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343271687 0 0) 0x7f86801970e0 con 0x7f8680100650 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f866c006640 con 0x7f8680100650 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2066033321 0 0) 0x7f866c006640 con 0x7f8680100650 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86801970e0 con 0x7f8680100650 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f8670001480 con 0x7f8680100650 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1972226720 0 0) 0x7f86801970e0 con 0x7f8680100650 2024-03-26T21:16:59.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86801972b0 con 0x7f8680100650 2024-03-26T21:16:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86801975a0 con 0x7f8680100650 2024-03-26T21:16:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f868019b0f0 con 0x7f8680100650 2024-03-26T21:16:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f86700036e0 con 0x7f8680100650 2024-03-26T21:16:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.225+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f8670004d60 con 0x7f8680100650 2024-03-26T21:16:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.226+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f8670004fe0 con 0x7f8680100650 2024-03-26T21:16:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.226+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8660005150 con 0x7f8680100650 2024-03-26T21:16:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.226+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7f8670047f80 con 0x7f8680100650 2024-03-26T21:16:59.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.231+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f8670017730 con 0x7f8680100650 2024-03-26T21:16:59.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.426+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 --> v1:172.21.15.88:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f8660005440 con 0x7f8680100650 2024-03-26T21:16:59.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.429+0000 7f867d7fa700 1 -- 172.21.15.88:0/283539009 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (unknown 3028693289 0 0) 0x7f8670017030 con 0x7f8680100650 2024-03-26T21:16:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.431+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 >> v1:172.21.15.88:6800/4097082509 conn(0x7f866c03a750 legacy=0x7f866c03cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.431+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 >> v1:172.21.15.88:6789/0 conn(0x7f8680100650 legacy=0x7f86801969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:16:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.431+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 shutdown_connections 2024-03-26T21:16:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.431+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 >> 172.21.15.88:0/283539009 conn(0x7f86800fbc10 msgr2=0x7f86800fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:16:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.431+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 shutdown_connections 2024-03-26T21:16:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:16:59.431+0000 7f8685dfd700 1 -- 172.21.15.88:0/283539009 wait complete. 2024-03-26T21:16:59.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:59.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:59.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:16:59.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:59.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:59.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:16:59.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/283539009' entity='client.admin' 2024-03-26T21:16:59.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:16:59.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:16:59 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:00.041 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T21:17:00.041 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T21:17:00.276 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:17:01.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.705+0000 7f7a67b60700 1 -- 172.21.15.88:0/2132867457 >> v1:172.21.15.88:6789/0 conn(0x7f7a60100e90 legacy=0x7f7a601012b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.705+0000 7f7a67b60700 1 -- 172.21.15.88:0/2132867457 shutdown_connections 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.705+0000 7f7a67b60700 1 -- 172.21.15.88:0/2132867457 >> 172.21.15.88:0/2132867457 conn(0x7f7a600fc430 msgr2=0x7f7a600fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.705+0000 7f7a67b60700 1 -- 172.21.15.88:0/2132867457 shutdown_connections 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.705+0000 7f7a67b60700 1 -- 172.21.15.88:0/2132867457 wait complete. 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a67b60700 1 Processor -- start 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a67b60700 1 -- start start 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a67b60700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a6019ca50 con 0x7f7a60100e90 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a658fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7a60100e90 0x7f7a600fe2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45560/0 (socket says 172.21.15.88:45560) 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a658fc700 1 -- 172.21.15.88:0/3295475873 learned_addr learned my addr 172.21.15.88:0/3295475873 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805444092 0 0) 0x7f7a6019ca50 con 0x7f7a60100e90 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a4c005530 con 0x7f7a60100e90 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 852130680 0 0) 0x7f7a4c005530 con 0x7f7a60100e90 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a6019ca50 con 0x7f7a60100e90 2024-03-26T21:17:01.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f7a50004750 con 0x7f7a60100e90 2024-03-26T21:17:01.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3743328701 0 0) 0x7f7a6019ca50 con 0x7f7a60100e90 2024-03-26T21:17:01.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a6019dc30 con 0x7f7a60100e90 2024-03-26T21:17:01.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a6019cc80 con 0x7f7a60100e90 2024-03-26T21:17:01.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.706+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a6019d1c0 con 0x7f7a60100e90 2024-03-26T21:17:01.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.707+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f7a50001190 con 0x7f7a60100e90 2024-03-26T21:17:01.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.707+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f7a50004e20 con 0x7f7a60100e90 2024-03-26T21:17:01.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.707+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f7a50004fe0 con 0x7f7a60100e90 2024-03-26T21:17:01.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.707+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a44005150 con 0x7f7a60100e90 2024-03-26T21:17:01.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.707+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7f7a50047d20 con 0x7f7a60100e90 2024-03-26T21:17:01.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.712+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f7a50017690 con 0x7f7a60100e90 2024-03-26T21:17:01.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.914+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f7a44000bc0 con 0x7f7a4c03a780 2024-03-26T21:17:01.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.918+0000 7f7a56ffd700 1 -- 172.21.15.88:0/3295475873 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f7a44000bc0 con 0x7f7a4c03a780 2024-03-26T21:17:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.920+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 >> v1:172.21.15.88:6800/4097082509 conn(0x7f7a4c03a780 legacy=0x7f7a4c03cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.920+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 >> v1:172.21.15.88:6789/0 conn(0x7f7a60100e90 legacy=0x7f7a600fe2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.920+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 shutdown_connections 2024-03-26T21:17:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.920+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 >> 172.21.15.88:0/3295475873 conn(0x7f7a600fc430 msgr2=0x7f7a60103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.920+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 shutdown_connections 2024-03-26T21:17:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:01.920+0000 7f7a67b60700 1 -- 172.21.15.88:0/3295475873 wait complete. 2024-03-26T21:17:02.725 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi138 2024-03-26T21:17:02.726 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:17:02.726 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.conf 2024-03-26T21:17:02.744 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:17:02.744 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:17:02.802 INFO:tasks.cephadm:Adding host smithi138 to orchestrator... 2024-03-26T21:17:02.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch host add smithi138 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='client.14174 v1:172.21.15.88:0/3295475873' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:02.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:02.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:03.037 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:17:04.079 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:03 smithi088 ceph-mon[30075]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:17:04.079 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:03 smithi088 ceph-mon[30075]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:17:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.525+0000 7f709127d700 1 -- 172.21.15.88:0/4228399909 >> v1:172.21.15.88:6789/0 conn(0x7f708c100700 legacy=0x7f708c100ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.525+0000 7f709127d700 1 -- 172.21.15.88:0/4228399909 shutdown_connections 2024-03-26T21:17:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.525+0000 7f709127d700 1 -- 172.21.15.88:0/4228399909 >> 172.21.15.88:0/4228399909 conn(0x7f708c0fc0c0 msgr2=0x7f708c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.525+0000 7f709127d700 1 -- 172.21.15.88:0/4228399909 shutdown_connections 2024-03-26T21:17:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.525+0000 7f709127d700 1 -- 172.21.15.88:0/4228399909 wait complete. 2024-03-26T21:17:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f709127d700 1 Processor -- start 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f709127d700 1 -- start start 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f709127d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f708c103570 con 0x7f708c100700 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f708affd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f708c100700 0x7f708c101680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40220/0 (socket says 172.21.15.88:40220) 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f708affd700 1 -- 172.21.15.88:0/1164215980 learned_addr learned my addr 172.21.15.88:0/1164215980 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737214992 0 0) 0x7f708c103570 con 0x7f708c100700 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7074005530 con 0x7f708c100700 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.526+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2442955293 0 0) 0x7f7074005530 con 0x7f708c100700 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f708c103570 con 0x7f708c100700 2024-03-26T21:17:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f707c001480 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1787736726 0 0) 0x7f708c103570 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c101d90 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f708c102080 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f708c1a1f00 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f707c003740 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.527+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f707c004d00 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.528+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f707c004ec0 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.528+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7078005120 con 0x7f708c100700 2024-03-26T21:17:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.528+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7f707c047bc0 con 0x7f708c100700 2024-03-26T21:17:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.533+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f707c0174c0 con 0x7f708c100700 2024-03-26T21:17:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:04.735+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}) -- 0x7f7078000b90 con 0x7f707403a890 2024-03-26T21:17:05.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:05 smithi088 ceph-mon[30075]: from='client.14176 v1:172.21.15.88:0/1164215980' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:06.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:06 smithi088 ceph-mon[30075]: Deploying cephadm binary to smithi138 2024-03-26T21:17:07.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:07 smithi088 ceph-mon[30075]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:08.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:09.004+0000 7f7073fff700 1 -- 172.21.15.88:0/1164215980 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (unknown 0 0 1185576259) 0x7f7078000b90 con 0x7f707403a890 2024-03-26T21:17:08.998 INFO:teuthology.orchestra.run.smithi088.stdout:Added host 'smithi138' with addr '172.21.15.138' 2024-03-26T21:17:09.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:09.006+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 >> v1:172.21.15.88:6800/4097082509 conn(0x7f707403a890 legacy=0x7f707403cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:09.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:09.006+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 >> v1:172.21.15.88:6789/0 conn(0x7f708c100700 legacy=0x7f708c101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:09.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:09.006+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 shutdown_connections 2024-03-26T21:17:09.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:09.006+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 >> 172.21.15.88:0/1164215980 conn(0x7f708c0fc0c0 msgr2=0x7f708c108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:09.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:09.007+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 shutdown_connections 2024-03-26T21:17:09.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:09.007+0000 7f709127d700 1 -- 172.21.15.88:0/1164215980 wait complete. 2024-03-26T21:17:09.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T21:17:09.935 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:17:10.312 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:10.313 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:10 smithi088 ceph-mon[30075]: Added host smithi138 2024-03-26T21:17:10.313 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:10.313 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:10 smithi088 ceph-mon[30075]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:10.313 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:11.046 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:11 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.527+0000 7fe90df78700 1 -- 172.21.15.88:0/3808705736 >> v1:172.21.15.88:6789/0 conn(0x7fe9081021a0 legacy=0x7fe908102580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.527+0000 7fe90df78700 1 -- 172.21.15.88:0/3808705736 shutdown_connections 2024-03-26T21:17:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.527+0000 7fe90df78700 1 -- 172.21.15.88:0/3808705736 >> 172.21.15.88:0/3808705736 conn(0x7fe9080fda70 msgr2=0x7fe9080ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.527+0000 7fe90df78700 1 -- 172.21.15.88:0/3808705736 shutdown_connections 2024-03-26T21:17:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.527+0000 7fe90df78700 1 -- 172.21.15.88:0/3808705736 wait complete. 2024-03-26T21:17:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.528+0000 7fe90df78700 1 Processor -- start 2024-03-26T21:17:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.528+0000 7fe90df78700 1 -- start start 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.528+0000 7fe90df78700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe9080767e0 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.528+0000 7fe9077fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe9081021a0 0x7fe908074940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40242/0 (socket says 172.21.15.88:40242) 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.528+0000 7fe9077fe700 1 -- 172.21.15.88:0/2615370203 learned_addr learned my addr 172.21.15.88:0/2615370203 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.528+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783002054 0 0) 0x7fe9080767e0 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.528+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8f0005530 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1082386840 0 0) 0x7fe8f0005530 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9080767e0 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fe8f8004700 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4165611525 0 0) 0x7fe9080767e0 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe908075050 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe908075340 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9081a1ee0 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fe8f8001ca0 con 0x7fe9081021a0 2024-03-26T21:17:11.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.529+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7fe8f8004fd0 con 0x7fe9081021a0 2024-03-26T21:17:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.530+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe908103680 con 0x7fe9081021a0 2024-03-26T21:17:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.530+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7fe8f8005290 con 0x7fe9081021a0 2024-03-26T21:17:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.530+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7fe8f8047d60 con 0x7fe9081021a0 2024-03-26T21:17:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.535+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fe8f80175b0 con 0x7fe9081021a0 2024-03-26T21:17:11.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.749+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe90810cbf0 con 0x7fe8f003a7a0 2024-03-26T21:17:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.750+0000 7fe904ff9700 1 -- 172.21.15.88:0/2615370203 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (unknown 0 0 634806500) 0x7fe90810cbf0 con 0x7fe8f003a7a0 2024-03-26T21:17:11.745 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:17:11.745 INFO:teuthology.orchestra.run.smithi088.stdout:[{"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}, {"addr": "172.21.15.138", "hostname": "smithi138", "labels": [], "status": ""}] 2024-03-26T21:17:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.752+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 >> v1:172.21.15.88:6800/4097082509 conn(0x7fe8f003a7a0 legacy=0x7fe8f003cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.752+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 >> v1:172.21.15.88:6789/0 conn(0x7fe9081021a0 legacy=0x7fe908074940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.752+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 shutdown_connections 2024-03-26T21:17:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.752+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 >> 172.21.15.88:0/2615370203 conn(0x7fe9080fda70 msgr2=0x7fe908108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.753+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 shutdown_connections 2024-03-26T21:17:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:11.753+0000 7fe90df78700 1 -- 172.21.15.88:0/2615370203 wait complete. 2024-03-26T21:17:12.330 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:12 smithi088 ceph-mon[30075]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:12.330 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:12 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:12.357 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T21:17:12.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T21:17:12.593 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:17:13.477 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:13 smithi088 ceph-mon[30075]: from='client.14178 v1:172.21.15.88:0/2615370203' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:17:14.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.266+0000 7ff50254f700 1 -- 172.21.15.88:0/1749363660 >> v1:172.21.15.88:6789/0 conn(0x7ff4fc101090 legacy=0x7ff4fc101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:14.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.266+0000 7ff50254f700 1 -- 172.21.15.88:0/1749363660 shutdown_connections 2024-03-26T21:17:14.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.266+0000 7ff50254f700 1 -- 172.21.15.88:0/1749363660 >> 172.21.15.88:0/1749363660 conn(0x7ff4fc0fc960 msgr2=0x7ff4fc0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:14.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.266+0000 7ff50254f700 1 -- 172.21.15.88:0/1749363660 shutdown_connections 2024-03-26T21:17:14.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.266+0000 7ff50254f700 1 -- 172.21.15.88:0/1749363660 wait complete. 2024-03-26T21:17:14.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.266+0000 7ff50254f700 1 Processor -- start 2024-03-26T21:17:14.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff50254f700 1 -- start start 2024-03-26T21:17:14.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff50254f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4fc198240 con 0x7ff4fc101090 2024-03-26T21:17:14.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff4fbfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff4fc101090 0x7ff4fc197b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39218/0 (socket says 172.21.15.88:39218) 2024-03-26T21:17:14.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff4fbfff700 1 -- 172.21.15.88:0/3519562535 learned_addr learned my addr 172.21.15.88:0/3519562535 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:17:14.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2074592454 0 0) 0x7ff4fc198240 con 0x7ff4fc101090 2024-03-26T21:17:14.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4e4005530 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2633405004 0 0) 0x7ff4e4005530 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4fc198240 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.267+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ff4f0004700 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.268+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2658267180 0 0) 0x7ff4fc198240 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.268+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4fc198410 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.268+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4fc198700 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.268+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4fc19fd70 con 0x7ff4fc101090 2024-03-26T21:17:14.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.268+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7ff4f0003760 con 0x7ff4fc101090 2024-03-26T21:17:14.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.268+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7ff4f0004d00 con 0x7ff4fc101090 2024-03-26T21:17:14.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.268+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7ff4f0004f80 con 0x7ff4fc101090 2024-03-26T21:17:14.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.269+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e8005120 con 0x7ff4fc101090 2024-03-26T21:17:14.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.269+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 102396990 0 0) 0x7ff4f0047db0 con 0x7ff4fc101090 2024-03-26T21:17:14.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.274+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7ff4f0017670 con 0x7ff4fc101090 2024-03-26T21:17:14.268 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:14 smithi088 ceph-mon[30075]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:14.475+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff4e8005410 con 0x7ff4fc101090 2024-03-26T21:17:15.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:15.069+0000 7ff4f97fa700 1 -- 172.21.15.88:0/3519562535 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (unknown 3126668360 0 0) 0x7ff4f0001ca0 con 0x7ff4fc101090 2024-03-26T21:17:15.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:15.071+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 >> v1:172.21.15.88:6800/4097082509 conn(0x7ff4e403a750 legacy=0x7ff4e403cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:15.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:15.071+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 >> v1:172.21.15.88:6789/0 conn(0x7ff4fc101090 legacy=0x7ff4fc197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:15.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:15.072+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 shutdown_connections 2024-03-26T21:17:15.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:15.072+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 >> 172.21.15.88:0/3519562535 conn(0x7ff4fc0fc960 msgr2=0x7ff4fc107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:15.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:15.072+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 shutdown_connections 2024-03-26T21:17:15.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:15.072+0000 7ff50254f700 1 -- 172.21.15.88:0/3519562535 wait complete. 2024-03-26T21:17:15.066 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2024-03-26T21:17:15.393 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3519562535' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T21:17:15.393 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:15.393 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:15.393 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:15.394 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:15.394 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:17:15.394 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:15.394 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:15.670 INFO:tasks.cephadm:Adding mon.a on smithi088 2024-03-26T21:17:15.670 INFO:tasks.cephadm:Adding mon.c on smithi088 2024-03-26T21:17:15.670 INFO:tasks.cephadm:Adding mon.b on smithi138 2024-03-26T21:17:15.670 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b' 2024-03-26T21:17:15.900 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3519562535' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:16 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.458+0000 7feac3d70700 1 -- 172.21.15.138:0/3261917431 >> v1:172.21.15.88:6789/0 conn(0x7feabc100730 legacy=0x7feabc100b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.458+0000 7feac3d70700 1 -- 172.21.15.138:0/3261917431 shutdown_connections 2024-03-26T21:17:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.458+0000 7feac3d70700 1 -- 172.21.15.138:0/3261917431 >> 172.21.15.138:0/3261917431 conn(0x7feabc0fc0d0 msgr2=0x7feabc0fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.458+0000 7feac3d70700 1 -- 172.21.15.138:0/3261917431 shutdown_connections 2024-03-26T21:17:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.458+0000 7feac3d70700 1 -- 172.21.15.138:0/3261917431 wait complete. 2024-03-26T21:17:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.459+0000 7feac3d70700 1 Processor -- start 2024-03-26T21:17:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.459+0000 7feac3d70700 1 -- start start 2024-03-26T21:17:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.459+0000 7feac3d70700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feabc197440 con 0x7feabc100730 2024-03-26T21:17:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.459+0000 7feac1b0c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feabc100730 0x7feabc196d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.138:38638/0 (socket says 172.21.15.138:38638) 2024-03-26T21:17:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.459+0000 7feac1b0c700 1 -- 172.21.15.138:0/3963987571 learned_addr learned my addr 172.21.15.138:0/3963987571 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:17:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1297297402 0 0) 0x7feabc197440 con 0x7feabc100730 2024-03-26T21:17:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaa8005530 con 0x7feabc100730 2024-03-26T21:17:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3000496117 0 0) 0x7feaa8005530 con 0x7feabc100730 2024-03-26T21:17:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feabc197440 con 0x7feabc100730 2024-03-26T21:17:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7feab8001480 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1085628249 0 0) 0x7feabc197440 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feabc197610 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feabc197900 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feabc19fb40 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7feab8003740 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.460+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7feab8004d80 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.461+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa0005120 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.461+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7feab8005000 con 0x7feabc100730 2024-03-26T21:17:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.461+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 396798714 0 0) 0x7feab8047e50 con 0x7feabc100730 2024-03-26T21:17:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.467+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7feab8078c30 con 0x7feabc100730 2024-03-26T21:17:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:17 smithi088 ceph-mon[30075]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:17:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:17 smithi088 ceph-mon[30075]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:17:17.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.674+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b", "target": ["mon-mgr", ""]}) -- 0x7feaa0000c60 con 0x7feaa803a7f0 2024-03-26T21:17:17.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.681+0000 7feab2ffd700 1 -- 172.21.15.138:0/3963987571 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7feaa0000c60 con 0x7feaa803a7f0 2024-03-26T21:17:17.675 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mon update... 2024-03-26T21:17:17.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.683+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 >> v1:172.21.15.88:6800/4097082509 conn(0x7feaa803a7f0 legacy=0x7feaa803ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:17.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.683+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 >> v1:172.21.15.88:6789/0 conn(0x7feabc100730 legacy=0x7feabc196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:17.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.683+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 shutdown_connections 2024-03-26T21:17:17.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.683+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 >> 172.21.15.138:0/3963987571 conn(0x7feabc0fc0d0 msgr2=0x7feabc108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:17.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.683+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 shutdown_connections 2024-03-26T21:17:17.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:17.683+0000 7feac3d70700 1 -- 172.21.15.138:0/3963987571 wait complete. 2024-03-26T21:17:18.287 DEBUG:teuthology.orchestra.run.smithi088:mon.c> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T21:17:18.290 DEBUG:teuthology.orchestra.run.smithi138:mon.b> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T21:17:18.293 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T21:17:18.293 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:17:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:18.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:18.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:19 smithi088 ceph-mon[30075]: from='client.14182 v1:172.21.15.138:0/3963987571' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:19 smithi088 ceph-mon[30075]: Saving service mon spec with placement smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b;count:3 2024-03-26T21:17:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:19 smithi088 ceph-mon[30075]: Deploying daemon mon.b on smithi138 2024-03-26T21:17:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:20 smithi088 ceph-mon[30075]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:21.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:21 smithi138 systemd[1]: Starting Ceph mon.b for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:17:22.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.115+0000 7f3f83012700 1 -- 172.21.15.138:0/719681312 >> v1:172.21.15.88:6789/0 conn(0x7f3f7c101090 legacy=0x7f3f7c101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:22.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.116+0000 7f3f83012700 1 -- 172.21.15.138:0/719681312 shutdown_connections 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.116+0000 7f3f83012700 1 -- 172.21.15.138:0/719681312 >> 172.21.15.138:0/719681312 conn(0x7f3f7c0fc900 msgr2=0x7f3f7c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.116+0000 7f3f83012700 1 -- 172.21.15.138:0/719681312 shutdown_connections 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.116+0000 7f3f83012700 1 -- 172.21.15.138:0/719681312 wait complete. 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.116+0000 7f3f83012700 1 Processor -- start 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.116+0000 7f3f83012700 1 -- start start 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.116+0000 7f3f83012700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f7c1995f0 con 0x7f3f7c101090 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f80dae700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3f7c101090 0x7f3f7c198ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.138:57478/0 (socket says 172.21.15.138:57478) 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f80dae700 1 -- 172.21.15.138:0/113880467 learned_addr learned my addr 172.21.15.138:0/113880467 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955953503 0 0) 0x7f3f7c1995f0 con 0x7f3f7c101090 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6c005530 con 0x7f3f7c101090 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1837692334 0 0) 0x7f3f6c005530 con 0x7f3f7c101090 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f7c1995f0 con 0x7f3f7c101090 2024-03-26T21:17:22.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3f70004700 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1540988300 0 0) 0x7f3f7c1995f0 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7c1997c0 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f7c199ab0 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f3f70003e60 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.117+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 1272412332 0 0) 0x7f3f70005220 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.118+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f7c1a1c70 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.118+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f3f700053e0 con 0x7f3f7c101090 2024-03-26T21:17:22.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.118+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7c063c70 con 0x7f3f7c101090 2024-03-26T21:17:22.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.118+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 396798714 0 0) 0x7f3f70047f60 con 0x7f3f7c101090 2024-03-26T21:17:22.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.124+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f3f7001f380 con 0x7f3f7c101090 2024-03-26T21:17:22.406 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 podman[33656]: 2024-03-26 21:17:22.280901156 +0000 UTC m=+0.102206054 container create 985d4c5f9dcb5251e5704a968ce3eb6b49da37de7ff70a42a09eeb2d8a101792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:17:22.406 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 podman[33656]: 2024-03-26 21:17:22.198501127 +0000 UTC m=+0.019806035 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:17:22.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.422+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3f7c10bbb0 con 0x7f3f7c101090 2024-03-26T21:17:22.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.423+0000 7f3f79ffb700 1 -- 172.21.15.138:0/113880467 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+694 (unknown 2237029548 0 1774324736) 0x7f3f70001180 con 0x7f3f7c101090 2024-03-26T21:17:22.417 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:17:22.417 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":1,"fsid":"e240a1a4-ebb5-11ee-95d0-87774f69a715","modified":"2024-03-26T21:15:43.819826Z","created":"2024-03-26T21:15:43.819826Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:17:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.425+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 >> v1:172.21.15.88:6800/4097082509 conn(0x7f3f6c03a750 legacy=0x7f3f6c03cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.425+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 >> v1:172.21.15.88:6789/0 conn(0x7f3f7c101090 legacy=0x7f3f7c198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.425+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 shutdown_connections 2024-03-26T21:17:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.425+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 >> 172.21.15.138:0/113880467 conn(0x7f3f7c0fc900 msgr2=0x7f3f7c06a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.425+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 shutdown_connections 2024-03-26T21:17:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:22.425+0000 7f3f83012700 1 -- 172.21.15.138:0/113880467 wait complete. 2024-03-26T21:17:22.420 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 1 2024-03-26T21:17:22.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:22 smithi088 ceph-mon[30075]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 podman[33656]: 2024-03-26 21:17:22.517031328 +0000 UTC m=+0.338336229 container init 985d4c5f9dcb5251e5704a968ce3eb6b49da37de7ff70a42a09eeb2d8a101792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 podman[33656]: 2024-03-26 21:17:22.522474258 +0000 UTC m=+0.343779157 container start 985d4c5f9dcb5251e5704a968ce3eb6b49da37de7ff70a42a09eeb2d8a101792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev), process ceph-mon, pid 7 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: pidfile_write: ignore empty --pid-file 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: load: jerasure load: lrc 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: RocksDB version: 7.9.2 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Git sha 0 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Compile date 2024-03-26 11:42:14 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: DB SUMMARY 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: DB Session ID: D4ADWBK4WLPO0J29X3CU 2024-03-26T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: CURRENT file: CURRENT 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.error_if_exists: 0 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.create_if_missing: 0 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.paranoid_checks: 1 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.env: 0x560e3ccc0c60 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.info_log: 0x560e3d761b40 2024-03-26T21:17:22.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T21:17:22.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.statistics: (nil) 2024-03-26T21:17:22.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.use_fsync: 0 2024-03-26T21:17:22.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_log_file_size: 0 2024-03-26T21:17:22.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T21:17:22.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T21:17:22.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.allow_fallocate: 1 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.use_direct_reads: 0 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.db_log_dir: 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.wal_dir: 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T21:17:22.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.write_buffer_manager: 0x560e3d7321e0 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T21:17:22.730 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.unordered_write: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.row_cache: None 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.wal_filter: None 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.two_write_queues: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.wal_compression: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.atomic_flush: 0 2024-03-26T21:17:22.731 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.log_readahead_size: 0 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_background_jobs: 2 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_background_compactions: -1 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_subcompactions: 1 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T21:17:22.732 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_open_files: -1 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_background_flushes: -1 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Compression algorithms supported: 2024-03-26T21:17:22.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kZSTD supported: 0 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kXpressCompression supported: 0 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kZlibCompression supported: 1 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kSnappyCompression supported: 1 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kLZ4Compression supported: 1 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: kBZip2Compression supported: 0 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.merge_operator: 2024-03-26T21:17:22.734 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_filter: None 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560e3d761d20) 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: cache_index_and_filter_blocks: 1 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: pin_top_level_index_and_filter: 1 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: index_type: 0 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: data_block_index_type: 0 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: index_shortening: 1 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: checksum: 4 2024-03-26T21:17:22.735 INFO:journalctl@ceph.mon.b.smithi138.stdout: no_block_cache: 0 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_cache: 0x560e3d765090 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_cache_name: BinnedLRUCache 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_cache_options: 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: capacity : 536870912 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: num_shard_bits : 4 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: strict_capacity_limit : 0 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: high_pri_pool_ratio: 0.000 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_cache_compressed: (nil) 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: persistent_cache: (nil) 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_size: 4096 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_size_deviation: 10 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_restart_interval: 16 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: index_block_restart_interval: 1 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: metadata_block_size: 4096 2024-03-26T21:17:22.736 INFO:journalctl@ceph.mon.b.smithi138.stdout: partition_filters: 0 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: use_delta_encoding: 1 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: filter_policy: bloomfilter 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: whole_key_filtering: 1 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: verify_compression: 0 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: read_amp_bytes_per_bit: 0 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: format_version: 5 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: enable_index_compression: 1 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: block_align: 0 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: max_auto_readahead_size: 262144 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: prepopulate_block_cache: 0 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: initial_auto_readahead_size: 8192 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T21:17:22.737 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression: NoCompression 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.num_levels: 7 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T21:17:22.738 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T21:17:22.739 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T21:17:22.740 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T21:17:22.741 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.inplace_update_support: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.bloom_locality: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.max_successive_merges: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.ttl: 2592000 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T21:17:22.742 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.enable_blob_files: false 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.min_blob_size: 0 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dd9a8c7d-2338-4066-a5fb-6ca425347c24 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487842551716, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T21:17:22.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487842552136, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711487842, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dd9a8c7d-2338-4066-a5fb-6ca425347c24", "db_session_id": "D4ADWBK4WLPO0J29X3CU", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487842552216, "job": 1, "event": "recovery_finished"} 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560e3d854000 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: DB pointer 0x560e3d83e000 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout: ** DB Stats ** 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:17:22.744 INFO:journalctl@ceph.mon.b.smithi138.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: ** Compaction Stats [default] ** 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: ** Compaction Stats [default] ** 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:22.745 INFO:journalctl@ceph.mon.b.smithi138.stdout: 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Block cache BinnedLRUCache@0x560e3d765090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T21:17:22.746 INFO:journalctl@ceph.mon.b.smithi138.stdout: 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: using public_addrv v1:172.21.15.138:6789/0 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: starting mon.b rank -1 at public addrs v1:172.21.15.138:6789/0 at bind addrs v1:172.21.15.138:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(???) e0 preinit fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout: e1 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout: legacy client fscid: -1 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout: 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout: No filesystems configured 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T21:17:22.747 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='client.14178 v1:172.21.15.88:0/2615370203' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3519562535' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.748 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3519562535' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:17:22.749 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: from='client.14182 v1:172.21.15.138:0/3963987571' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: Saving service mon spec with placement smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b;count:3 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: Deploying daemon mon.b on smithi138 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:22.750 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 ceph-mon[33706]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T21:17:22.751 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 bash[33656]: 985d4c5f9dcb5251e5704a968ce3eb6b49da37de7ff70a42a09eeb2d8a101792 2024-03-26T21:17:22.751 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:22 smithi138 systemd[1]: Started Ceph mon.b for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:17:24.199 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T21:17:24.200 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:17:24.453 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.630+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/3133801537 >> v1:172.21.15.88:6789/0 conn(0x7f5ff01011f0 legacy=0x7f5ff01015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.630+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/3133801537 shutdown_connections 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.630+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/3133801537 >> 172.21.15.138:0/3133801537 conn(0x7f5ff00fca60 msgr2=0x7f5ff00fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.630+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/3133801537 shutdown_connections 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.630+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/3133801537 wait complete. 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.631+0000 7f5ff7ab2700 1 Processor -- start 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.631+0000 7f5ff7ab2700 1 -- start start 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.632+0000 7f5ff7ab2700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ff01a0f00 con 0x7f5ff01011f0 2024-03-26T21:17:27.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.632+0000 7f5ff7ab2700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ff01a20e0 con 0x7f5ff019b150 2024-03-26T21:17:27.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.632+0000 7f5ff504d700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f5ff019b150 0x7f5ff019f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47430/0 (socket says 172.21.15.138:47430) 2024-03-26T21:17:27.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.632+0000 7f5ff504d700 1 -- 172.21.15.138:0/155247945 learned_addr learned my addr 172.21.15.138:0/155247945 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3884702142 0 0) 0x7f5ff01a0f00 con 0x7f5ff01011f0 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fdc0035b0 con 0x7f5ff01011f0 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2225377833 0 0) 0x7f5fdc0035b0 con 0x7f5ff01011f0 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ff01a0f00 con 0x7f5ff01011f0 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3556990546 0 0) 0x7f5fec001500 con 0x7f5ff01011f0 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1377354420 0 0) 0x7f5ff01a0f00 con 0x7f5ff01011f0 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 >> v1:172.21.15.138:6789/0 conn(0x7f5ff019b150 legacy=0x7f5ff019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff01a32c0 con 0x7f5ff01011f0 2024-03-26T21:17:27.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff01a1130 con 0x7f5ff01011f0 2024-03-26T21:17:27.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ff01a1630 con 0x7f5ff01011f0 2024-03-26T21:17:27.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f5fec003a10 con 0x7f5ff01011f0 2024-03-26T21:17:27.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.633+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3556990546 0 0) 0x7f5fec004ef0 con 0x7f5ff01011f0 2024-03-26T21:17:27.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.634+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f5fec010630 con 0x7f5ff01011f0 2024-03-26T21:17:27.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.634+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd4005120 con 0x7f5ff01011f0 2024-03-26T21:17:27.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.635+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 396798714 0 0) 0x7f5fec047ae0 con 0x7f5ff01011f0 2024-03-26T21:17:27.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.640+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f5fec016440 con 0x7f5ff01011f0 2024-03-26T21:17:27.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.921+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5fd4005410 con 0x7f5ff01011f0 2024-03-26T21:17:27.916 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:17:27.916 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: mon.a calling monitor election 2024-03-26T21:17:27.916 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.916 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:27.916 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.916 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.916 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: mon.b calling monitor election 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: monmap epoch 2 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: last_changed 2024-03-26T21:17:22.573122+0000 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: min_mon_release 19 (squid) 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: election_strategy: 1 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: 1: v1:172.21.15.138:6789/0 mon.b 2024-03-26T21:17:27.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: fsmap 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: mgrmap e13: y(active, since 40s) 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: overall HEALTH_OK 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:27.918 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:27.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.922+0000 7f5fe6ffd700 1 -- 172.21.15.138:0/155247945 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+912 (unknown 3557084514 0 1832174443) 0x7f5fec015d40 con 0x7f5ff01011f0 2024-03-26T21:17:27.921 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:17:27.921 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":2,"fsid":"e240a1a4-ebb5-11ee-95d0-87774f69a715","modified":"2024-03-26T21:17:22.573122Z","created":"2024-03-26T21:15:43.819826Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T21:17:27.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.925+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 >> v1:172.21.15.88:6800/4097082509 conn(0x7f5fdc03ca80 legacy=0x7f5fdc03ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:27.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.925+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 >> v1:172.21.15.88:6789/0 conn(0x7f5ff01011f0 legacy=0x7f5ff019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:27.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.925+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 shutdown_connections 2024-03-26T21:17:27.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.925+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 >> 172.21.15.138:0/155247945 conn(0x7f5ff00fca60 msgr2=0x7f5ff0108140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:27.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.925+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 shutdown_connections 2024-03-26T21:17:27.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:27.925+0000 7f5ff7ab2700 1 -- 172.21.15.138:0/155247945 wait complete. 2024-03-26T21:17:27.923 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 2 2024-03-26T21:17:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:17:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: mon.a calling monitor election 2024-03-26T21:17:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: mon.b calling monitor election 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: monmap epoch 2 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: last_changed 2024-03-26T21:17:22.573122+0000 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: min_mon_release 19 (squid) 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: election_strategy: 1 2024-03-26T21:17:27.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: 1: v1:172.21.15.138:6789/0 mon.b 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: fsmap 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: mgrmap e13: y(active, since 40s) 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: overall HEALTH_OK 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:27.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:28.922 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:28 smithi088 ceph-mon[30075]: Deploying daemon mon.c on smithi088 2024-03-26T21:17:28.923 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/155247945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:17:28.923 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:28 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:28 smithi138 ceph-mon[33706]: Deploying daemon mon.c on smithi088 2024-03-26T21:17:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/155247945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:17:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:28 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:29.512 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T21:17:29.512 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:17:29.759 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:17:29.881 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:29 smithi138 ceph-mon[33706]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:29.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:29 smithi088 ceph-mon[30075]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:29.954 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:17:29 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:17:29.572+0000 7f7d1c0d8700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-26T21:17:30.971 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:30 smithi088 podman[39164]: 2024-03-26 21:17:30.762400861 +0000 UTC m=+0.017658695 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:17:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 podman[39164]: 2024-03-26 21:17:31.04070928 +0000 UTC m=+0.295967094 container init a36647f7c14f3a2241fcce62acf2983ac7f2512c1e3f5db49bd4172221161966 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-26T21:17:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 podman[39164]: 2024-03-26 21:17:31.045938143 +0000 UTC m=+0.301195961 container start a36647f7c14f3a2241fcce62acf2983ac7f2512c1e3f5db49bd4172221161966 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux ) 2024-03-26T21:17:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev), process ceph-mon, pid 7 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pidfile_write: ignore empty --pid-file 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: load: jerasure load: lrc 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: RocksDB version: 7.9.2 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Git sha 0 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Compile date 2024-03-26 11:42:14 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: DB SUMMARY 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: DB Session ID: 46CWQ6C8XGADR6LSJRCV 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: CURRENT file: CURRENT 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-26T21:17:31.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.error_if_exists: 0 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.create_if_missing: 0 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.paranoid_checks: 1 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.env: 0x55f1858d6c60 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.info_log: 0x55f186681b00 2024-03-26T21:17:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.statistics: (nil) 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.use_fsync: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_log_file_size: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.allow_fallocate: 1 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.use_direct_reads: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T21:17:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.db_log_dir: 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.wal_dir: 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.write_buffer_manager: 0x55f1866521e0 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T21:17:31.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.unordered_write: 0 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.row_cache: None 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.wal_filter: None 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T21:17:31.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.two_write_queues: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.wal_compression: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.atomic_flush: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.log_readahead_size: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T21:17:31.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_background_jobs: 2 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_background_compactions: -1 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_subcompactions: 1 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_open_files: -1 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T21:17:31.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_background_flushes: -1 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Compression algorithms supported: 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kZSTD supported: 0 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kXpressCompression supported: 0 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kZlibCompression supported: 1 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kSnappyCompression supported: 1 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kLZ4Compression supported: 1 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: kBZip2Compression supported: 0 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T21:17:31.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.merge_operator: 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_filter: None 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f186681ce0) 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout: cache_index_and_filter_blocks: 1 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T21:17:31.494 INFO:journalctl@ceph.mon.c.smithi088.stdout: pin_top_level_index_and_filter: 1 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: index_type: 0 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: data_block_index_type: 0 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: index_shortening: 1 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: checksum: 4 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: no_block_cache: 0 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache: 0x55f186685090 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache_name: BinnedLRUCache 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache_options: 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: capacity : 536870912 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: num_shard_bits : 4 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: strict_capacity_limit : 0 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: high_pri_pool_ratio: 0.000 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache_compressed: (nil) 2024-03-26T21:17:31.495 INFO:journalctl@ceph.mon.c.smithi088.stdout: persistent_cache: (nil) 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_size: 4096 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_size_deviation: 10 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_restart_interval: 16 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: index_block_restart_interval: 1 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: metadata_block_size: 4096 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: partition_filters: 0 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: use_delta_encoding: 1 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: filter_policy: bloomfilter 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: whole_key_filtering: 1 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: verify_compression: 0 2024-03-26T21:17:31.496 INFO:journalctl@ceph.mon.c.smithi088.stdout: read_amp_bytes_per_bit: 0 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout: format_version: 5 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout: enable_index_compression: 1 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_align: 0 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout: max_auto_readahead_size: 262144 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout: prepopulate_block_cache: 0 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout: initial_auto_readahead_size: 8192 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression: NoCompression 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T21:17:31.497 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.num_levels: 7 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T21:17:31.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T21:17:31.499 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T21:17:31.500 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.inplace_update_support: 0 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T21:17:31.501 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.bloom_locality: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.max_successive_merges: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.ttl: 2592000 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.enable_blob_files: false 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.min_blob_size: 0 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T21:17:31.502 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5d6826fd-8b2d-4474-bd6d-3b74ef79d25e 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487851075607, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487851076069, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711487851, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5d6826fd-8b2d-4474-bd6d-3b74ef79d25e", "db_session_id": "46CWQ6C8XGADR6LSJRCV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711487851076158, "job": 1, "event": "recovery_finished"} 2024-03-26T21:17:31.503 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f186774000 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: DB pointer 0x55f18675e000 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** DB Stats ** 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:17:31.504 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** Compaction Stats [default] ** 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** Compaction Stats [default] ** 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-26T21:17:31.505 INFO:journalctl@ceph.mon.c.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: Block cache BinnedLRUCache@0x55f186685090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T21:17:31.506 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: using public_addrv v1:172.21.15.88:6790/0 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: starting mon.c rank -1 at public addrs v1:172.21.15.88:6790/0 at bind addrs v1:172.21.15.88:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(???) e0 preinit fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout: e1 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout: legacy client fscid: -1 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout: No filesystems configured 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T21:17:31.507 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='client.14178 v1:172.21.15.88:0/2615370203' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3519562535' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3519562535' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:17:31.509 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='client.14182 v1:172.21.15.138:0/3963987571' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: Saving service mon spec with placement smithi088:[v1:172.21.15.88:6789]=a;smithi088:[v1:172.21.15.88:6790]=c;smithi138:[v1:172.21.15.138:6789]=b;count:3 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: Deploying daemon mon.b on smithi138 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:17:31.510 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.a calling monitor election 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.b calling monitor election 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: monmap epoch 2 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: last_changed 2024-03-26T21:17:22.573122+0000 2024-03-26T21:17:31.511 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: min_mon_release 19 (squid) 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: election_strategy: 1 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: 1: v1:172.21.15.138:6789/0 mon.b 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: fsmap 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mgrmap e13: y(active, since 40s) 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: overall HEALTH_OK 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:31.512 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: Deploying daemon mon.c on smithi088 2024-03-26T21:17:31.513 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/155247945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:17:31.513 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:31.513 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:31.513 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 ceph-mon[39202]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-26T21:17:31.513 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 bash[39164]: a36647f7c14f3a2241fcce62acf2983ac7f2512c1e3f5db49bd4172221161966 2024-03-26T21:17:31.513 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:31 smithi088 systemd[1]: Started Ceph mon.c for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:17:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.143+0000 7f3d48162700 1 -- 172.21.15.138:0/3096231169 >> v1:172.21.15.88:6789/0 conn(0x7f3d240035f0 legacy=0x7f3d24005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.143+0000 7f3d48162700 1 -- 172.21.15.138:0/3096231169 shutdown_connections 2024-03-26T21:17:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.143+0000 7f3d48162700 1 -- 172.21.15.138:0/3096231169 >> 172.21.15.138:0/3096231169 conn(0x7f3d400fdbf0 msgr2=0x7f3d40100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.143+0000 7f3d48162700 1 -- 172.21.15.138:0/3096231169 shutdown_connections 2024-03-26T21:17:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.143+0000 7f3d48162700 1 -- 172.21.15.138:0/3096231169 wait complete. 2024-03-26T21:17:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.144+0000 7f3d48162700 1 Processor -- start 2024-03-26T21:17:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.144+0000 7f3d48162700 1 -- start start 2024-03-26T21:17:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.144+0000 7f3d48162700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3d401a9050 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.144+0000 7f3d48162700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3d401aa270 con 0x7f3d240035f0 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.144+0000 7f3d48162700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3d401ab4b0 con 0x7f3d401a5380 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.144+0000 7f3d45efe700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f3d240035f0 0x7f3d401990f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:55820/0 (socket says 172.21.15.138:55820) 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.144+0000 7f3d45efe700 1 -- 172.21.15.138:0/3668910419 learned_addr learned my addr 172.21.15.138:0/3668910419 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.145+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263580800 0 0) 0x7f3d401a9050 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.145+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d2c003bf0 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.145+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 91325119 0 0) 0x7f3d2c003bf0 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.145+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d401a9050 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.145+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3d30001760 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 884271538 0 0) 0x7f3d401a9050 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 >> v1:172.21.15.88:6790/0 conn(0x7f3d401a5380 legacy=0x7f3d401a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 >> v1:172.21.15.138:6789/0 conn(0x7f3d240035f0 legacy=0x7f3d401990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d401ac6f0 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d401ab6e0 con 0x7f3d40199800 2024-03-26T21:17:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d401abbe0 con 0x7f3d40199800 2024-03-26T21:17:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f3d30003b10 con 0x7f3d40199800 2024-03-26T21:17:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.146+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3d300049a0 con 0x7f3d40199800 2024-03-26T21:17:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.147+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f3d30010100 con 0x7f3d40199800 2024-03-26T21:17:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.147+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d24006b90 con 0x7f3d40199800 2024-03-26T21:17:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.147+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 396798714 0 0) 0x7f3d300477c0 con 0x7f3d40199800 2024-03-26T21:17:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.155+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f3d30016120 con 0x7f3d40199800 2024-03-26T21:17:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.448+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3d24007260 con 0x7f3d40199800 2024-03-26T21:17:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.448+0000 7f3d36ffd700 1 -- 172.21.15.138:0/3668910419 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1127 (unknown 409899127 0 3776194900) 0x7f3d30015a20 con 0x7f3d40199800 2024-03-26T21:17:36.442 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-26T21:17:36.443 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":3,"fsid":"e240a1a4-ebb5-11ee-95d0-87774f69a715","modified":"2024-03-26T21:17:31.094333Z","created":"2024-03-26T21:15:43.819826Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6790","nonce":0}]},"addr":"172.21.15.88:6790/0","public_addr":"172.21.15.88:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: mon.a calling monitor election 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: mon.b calling monitor election 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.443 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: mon.c calling monitor election 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: monmap epoch 3 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: last_changed 2024-03-26T21:17:31.094333+0000 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: min_mon_release 19 (squid) 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: election_strategy: 1 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: 1: v1:172.21.15.138:6789/0 mon.b 2024-03-26T21:17:36.444 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: 2: v1:172.21.15.88:6790/0 mon.c 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: fsmap 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: mgrmap e13: y(active, since 49s) 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: overall HEALTH_OK 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.445 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.451+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 >> v1:172.21.15.88:6800/4097082509 conn(0x7f3d2c03a6f0 legacy=0x7f3d2c03cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.452+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 >> v1:172.21.15.88:6789/0 conn(0x7f3d40199800 legacy=0x7f3d401a3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.452+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 shutdown_connections 2024-03-26T21:17:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.452+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 >> 172.21.15.138:0/3668910419 conn(0x7f3d400fdbf0 msgr2=0x7f3d400fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.452+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 shutdown_connections 2024-03-26T21:17:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:36.452+0000 7f3d48162700 1 -- 172.21.15.138:0/3668910419 wait complete. 2024-03-26T21:17:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 3 2024-03-26T21:17:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:17:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: mon.a calling monitor election 2024-03-26T21:17:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: mon.b calling monitor election 2024-03-26T21:17:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: mon.c calling monitor election 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: monmap epoch 3 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: last_changed 2024-03-26T21:17:31.094333+0000 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:17:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: min_mon_release 19 (squid) 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: election_strategy: 1 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: 1: v1:172.21.15.138:6789/0 mon.b 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: 2: v1:172.21.15.88:6790/0 mon.c 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: fsmap 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: mgrmap e13: y(active, since 49s) 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: overall HEALTH_OK 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: mon.a calling monitor election 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: mon.b calling monitor election 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: mon.c calling monitor election 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:36.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: monmap epoch 3 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: last_changed 2024-03-26T21:17:31.094333+0000 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: created 2024-03-26T21:15:43.819826+0000 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: min_mon_release 19 (squid) 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: election_strategy: 1 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: 0: v1:172.21.15.88:6789/0 mon.a 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: 1: v1:172.21.15.138:6789/0 mon.b 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: 2: v1:172.21.15.88:6790/0 mon.c 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: fsmap 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: mgrmap e13: y(active, since 49s) 2024-03-26T21:17:36.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: overall HEALTH_OK 2024-03-26T21:17:36.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:36.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:37.020 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T21:17:37.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T21:17:37.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:37.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/3668910419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:17:37.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:37.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/3668910419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:17:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/3668910419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:17:37.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:17:37.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:37.597 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:17:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:38 smithi138 ceph-mon[33706]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:38 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:38 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:38 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:38 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:38 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:38 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[30075]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[39202]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:38 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:39.320 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-26T21:17:39.320 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:17:39.320 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:39.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.323 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:39.323 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:39.323 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:39 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:39.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:39.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:39 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:39.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.549+0000 7fc321474700 1 -- 172.21.15.88:0/221939787 >> v1:172.21.15.88:6789/0 conn(0x7fc31c1010e0 legacy=0x7fc31c1014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:39.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.549+0000 7fc321474700 1 -- 172.21.15.88:0/221939787 shutdown_connections 2024-03-26T21:17:39.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.549+0000 7fc321474700 1 -- 172.21.15.88:0/221939787 >> 172.21.15.88:0/221939787 conn(0x7fc31c0fc990 msgr2=0x7fc31c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:39.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.549+0000 7fc321474700 1 -- 172.21.15.88:0/221939787 shutdown_connections 2024-03-26T21:17:39.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.549+0000 7fc321474700 1 -- 172.21.15.88:0/221939787 wait complete. 2024-03-26T21:17:39.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc321474700 1 Processor -- start 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc321474700 1 -- start start 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc321474700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc31c1a6df0 con 0x7fc31c1975a0 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc321474700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc31c1a8010 con 0x7fc31c1a3120 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc321474700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc31c1a9250 con 0x7fc31c1010e0 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc31a7fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc31c1975a0 0x7fc31c1a1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43290/0 (socket says 172.21.15.88:43290) 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc31a7fc700 1 -- 172.21.15.88:0/3143023038 learned_addr learned my addr 172.21.15.88:0/3143023038 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995890410 0 0) 0x7fc31c1a6df0 con 0x7fc31c1975a0 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.550+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc304003bf0 con 0x7fc31c1975a0 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 415504462 0 0) 0x7fc31c1a9250 con 0x7fc31c1010e0 2024-03-26T21:17:39.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc31c1a6df0 con 0x7fc31c1010e0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3166385962 0 0) 0x7fc304003bf0 con 0x7fc31c1975a0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc31c1a9250 con 0x7fc31c1975a0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc30c001760 con 0x7fc31c1975a0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065514061 0 0) 0x7fc31c1a8010 con 0x7fc31c1a3120 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc304003bf0 con 0x7fc31c1a3120 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 385283503 0 0) 0x7fc31c1a6df0 con 0x7fc31c1010e0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc31c1a8010 con 0x7fc31c1010e0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc310001940 con 0x7fc31c1010e0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1578859394 0 0) 0x7fc31c1a9250 con 0x7fc31c1975a0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 >> v1:172.21.15.88:6790/0 conn(0x7fc31c1010e0 legacy=0x7fc31c196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 >> v1:172.21.15.138:6789/0 conn(0x7fc31c1a3120 legacy=0x7fc31c1a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c1aa490 con 0x7fc31c1975a0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc31c1a7020 con 0x7fc31c1975a0 2024-03-26T21:17:39.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc31c1a74e0 con 0x7fc31c1975a0 2024-03-26T21:17:39.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fc30c001d10 con 0x7fc31c1975a0 2024-03-26T21:17:39.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.551+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc30c005bd0 con 0x7fc31c1975a0 2024-03-26T21:17:39.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.552+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7fc30c0113f0 con 0x7fc31c1975a0 2024-03-26T21:17:39.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.552+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc308005120 con 0x7fc31c1975a0 2024-03-26T21:17:39.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.552+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 396798714 0 0) 0x7fc30c047c20 con 0x7fc31c1975a0 2024-03-26T21:17:39.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.557+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fc30c017430 con 0x7fc31c1975a0 2024-03-26T21:17:39.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.758+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc308005410 con 0x7fc31c1975a0 2024-03-26T21:17:39.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.758+0000 7fc303fff700 1 -- 172.21.15.88:0/3143023038 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+191 (unknown 2189217731 0 3443614721) 0x7fc30c005550 con 0x7fc31c1975a0 2024-03-26T21:17:39.752 INFO:teuthology.orchestra.run.smithi088.stdout:# minimal ceph.conf for e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:39.752 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2024-03-26T21:17:39.752 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:17:39.752 INFO:teuthology.orchestra.run.smithi088.stdout: mon_host = 172.21.15.88:6789/0 172.21.15.138:6789/0 v1:172.21.15.88:6790/0 2024-03-26T21:17:39.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.760+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 >> v1:172.21.15.88:6800/4097082509 conn(0x7fc30403a840 legacy=0x7fc30403cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:39.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.760+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 >> v1:172.21.15.88:6789/0 conn(0x7fc31c1975a0 legacy=0x7fc31c1a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:39.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.760+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 shutdown_connections 2024-03-26T21:17:39.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.760+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 >> 172.21.15.88:0/3143023038 conn(0x7fc31c0fc990 msgr2=0x7fc31c1083e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:39.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.760+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 shutdown_connections 2024-03-26T21:17:39.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:39.760+0000 7fc321474700 1 -- 172.21.15.88:0/3143023038 wait complete. 2024-03-26T21:17:40.314 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[30075]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T21:17:40.314 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[30075]: Reconfiguring daemon mon.a on smithi088 2024-03-26T21:17:40.314 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[30075]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:40.314 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3143023038' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:40.314 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[39202]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T21:17:40.315 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[39202]: Reconfiguring daemon mon.a on smithi088 2024-03-26T21:17:40.315 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[39202]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:40.315 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3143023038' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:40 smithi138 ceph-mon[33706]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T21:17:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:40 smithi138 ceph-mon[33706]: Reconfiguring daemon mon.a on smithi088 2024-03-26T21:17:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:40 smithi138 ceph-mon[33706]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3143023038' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:40.615 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T21:17:40.616 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:17:40.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T21:17:40.645 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:17:40.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:17:40.714 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:17:40.714 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T21:17:40.750 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:17:40.750 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:17:40.820 INFO:tasks.cephadm:Adding mgr.y on smithi088 2024-03-26T21:17:40.820 INFO:tasks.cephadm:Adding mgr.x on smithi138 2024-03-26T21:17:40.820 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi088=y;smithi138=x' 2024-03-26T21:17:41.092 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:17:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: Reconfiguring mon.c (monmap changed)... 2024-03-26T21:17:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:42 smithi138 ceph-mon[33706]: Reconfiguring daemon mon.c on smithi088 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: Reconfiguring mon.c (monmap changed)... 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[30075]: Reconfiguring daemon mon.c on smithi088 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: Reconfiguring mon.c (monmap changed)... 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:42 smithi088 ceph-mon[39202]: Reconfiguring daemon mon.c on smithi088 2024-03-26T21:17:42.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.611+0000 7f08e9aad700 1 -- 172.21.15.138:0/3865266869 >> v1:172.21.15.88:6789/0 conn(0x7f08e41020e0 legacy=0x7f08e4102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:42.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.611+0000 7f08e9aad700 1 -- 172.21.15.138:0/3865266869 shutdown_connections 2024-03-26T21:17:42.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.611+0000 7f08e9aad700 1 -- 172.21.15.138:0/3865266869 >> 172.21.15.138:0/3865266869 conn(0x7f08e40fd6a0 msgr2=0x7f08e40ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:42.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.611+0000 7f08e9aad700 1 -- 172.21.15.138:0/3865266869 shutdown_connections 2024-03-26T21:17:42.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.612+0000 7f08e9aad700 1 -- 172.21.15.138:0/3865266869 wait complete. 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.612+0000 7f08e9aad700 1 Processor -- start 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.612+0000 7f08e9aad700 1 -- start start 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e9aad700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e41a2370 con 0x7f08e4197210 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e9aad700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e41a3590 con 0x7f08e419e700 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e9aad700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e41a47d0 con 0x7f08e41020e0 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e3fff700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f08e419e700 0x7f08e41a0af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34208/0 (socket says 172.21.15.138:34208) 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08dbfff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f08e4197210 0x7f08e419cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.138:45072/0 (socket says 172.21.15.138:45072) 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e3fff700 1 -- 172.21.15.138:0/2997408474 learned_addr learned my addr 172.21.15.138:0/2997408474 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:17:42.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352471325 0 0) 0x7f08e41a3590 con 0x7f08e419e700 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08d0003bf0 con 0x7f08e419e700 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 117426564 0 0) 0x7f08e41a2370 con 0x7f08e4197210 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08e41a3590 con 0x7f08e4197210 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621721448 0 0) 0x7f08e41a47d0 con 0x7f08e41020e0 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08e41a2370 con 0x7f08e41020e0 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 148545929 0 0) 0x7f08d0003bf0 con 0x7f08e419e700 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08e41a47d0 con 0x7f08e419e700 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f08dc001b70 con 0x7f08e419e700 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1808092991 0 0) 0x7f08e41a2370 con 0x7f08e41020e0 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08d0003bf0 con 0x7f08e41020e0 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3289093987 0 0) 0x7f08e41a3590 con 0x7f08e4197210 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08e41a2370 con 0x7f08e4197210 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f08d4003910 con 0x7f08e41020e0 2024-03-26T21:17:42.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f08cc001810 con 0x7f08e4197210 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1806340087 0 0) 0x7f08e41a47d0 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 >> v1:172.21.15.88:6790/0 conn(0x7f08e41020e0 legacy=0x7f08e4196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 >> v1:172.21.15.88:6789/0 conn(0x7f08e4197210 legacy=0x7f08e419cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e41a5a10 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.613+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08e41a25a0 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.614+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f08e41a2ae0 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.614+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f08dc0051c0 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.614+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f08dc006050 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.615+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f08dc011870 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.615+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c4005150 con 0x7f08e419e700 2024-03-26T21:17:42.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.615+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 396798714 0 0) 0x7f08dc047e70 con 0x7f08e419e700 2024-03-26T21:17:42.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.620+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f08dc0176c0 con 0x7f08e419e700 2024-03-26T21:17:42.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.834+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi088=y;smithi138=x", "target": ["mon-mgr", ""]}) -- 0x7f08c4000bc0 con 0x7f08d003a8b0 2024-03-26T21:17:42.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.839+0000 7f08e17fa700 1 -- 172.21.15.138:0/2997408474 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7f08c4000bc0 con 0x7f08d003a8b0 2024-03-26T21:17:42.833 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mgr update... 2024-03-26T21:17:42.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.843+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 >> v1:172.21.15.88:6800/4097082509 conn(0x7f08d003a8b0 legacy=0x7f08d003cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:42.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.843+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 >> v1:172.21.15.138:6789/0 conn(0x7f08e419e700 legacy=0x7f08e41a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:42.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.843+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 shutdown_connections 2024-03-26T21:17:42.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.843+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 >> 172.21.15.138:0/2997408474 conn(0x7f08e40fd6a0 msgr2=0x7f08e40ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:42.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.843+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 shutdown_connections 2024-03-26T21:17:42.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:17:42.843+0000 7f08e9aad700 1 -- 172.21.15.138:0/2997408474 wait complete. 2024-03-26T21:17:43.484 DEBUG:teuthology.orchestra.run.smithi138:mgr.x> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T21:17:43.487 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T21:17:43.487 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:17:43.487 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T21:17:43.507 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T21:17:43.507 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-03-26T21:17:43.564 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T21:17:43.564 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:43.564 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T21:17:43.564 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:43.564 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:43.564 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-26 21:17:30.584582188 +0000 2024-03-26T21:17:43.564 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-26 21:11:41.926289134 +0000 2024-03-26T21:17:43.565 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-26 21:11:41.926289134 +0000 2024-03-26T21:17:43.565 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-26 21:11:41.926289134 +0000 2024-03-26T21:17:43.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T21:17:43.635 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-26T21:17:43.635 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-26T21:17:43.635 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000286586 s, 1.8 MB/s 2024-03-26T21:17:43.636 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T21:17:43.694 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-03-26T21:17:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: Reconfiguring mon.b (monmap changed)... 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: Reconfiguring daemon mon.b on smithi138 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: from='client.24103 v1:172.21.15.138:0/2997408474' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi088=y;smithi138=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: Saving service mgr spec with placement smithi088=y;smithi138=x;count:2 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:43 smithi138 ceph-mon[33706]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:43.749 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T21:17:43.749 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:43.750 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T21:17:43.750 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:43.750 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:43.750 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-26 21:17:30.584582188 +0000 2024-03-26T21:17:43.750 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-26 21:11:42.262282277 +0000 2024-03-26T21:17:43.750 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-26 21:11:42.262282277 +0000 2024-03-26T21:17:43.750 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-26 21:11:42.262282277 +0000 2024-03-26T21:17:43.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T21:17:43.816 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-26T21:17:43.816 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-26T21:17:43.816 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000252404 s, 2.0 MB/s 2024-03-26T21:17:43.817 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T21:17:43.874 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-26 21:17:30.585582168 +0000 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-26 21:11:42.586275664 +0000 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-26 21:11:42.586275664 +0000 2024-03-26T21:17:43.930 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-26 21:11:42.586275664 +0000 2024-03-26T21:17:43.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: Reconfiguring mon.b (monmap changed)... 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: Reconfiguring daemon mon.b on smithi138 2024-03-26T21:17:43.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: from='client.24103 v1:172.21.15.138:0/2997408474' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi088=y;smithi138=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: Saving service mgr spec with placement smithi088=y;smithi138=x;count:2 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[30075]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: Reconfiguring mon.b (monmap changed)... 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:43.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: Reconfiguring daemon mon.b on smithi138 2024-03-26T21:17:43.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: from='client.24103 v1:172.21.15.138:0/2997408474' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi088=y;smithi138=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:43.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: Saving service mgr spec with placement smithi088=y;smithi138=x;count:2 2024-03-26T21:17:43.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:43.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:43 smithi088 ceph-mon[39202]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:43.996 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-26T21:17:43.997 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-26T21:17:43.997 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000197741 s, 2.6 MB/s 2024-03-26T21:17:43.998 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T21:17:44.053 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-26 21:17:30.585582168 +0000 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-26 21:11:42.909269072 +0000 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-26 21:11:42.909269072 +0000 2024-03-26T21:17:44.109 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-26 21:11:42.909269072 +0000 2024-03-26T21:17:44.109 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T21:17:44.175 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-26T21:17:44.176 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-26T21:17:44.176 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000176285 s, 2.9 MB/s 2024-03-26T21:17:44.177 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T21:17:44.232 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:17:44.232 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T21:17:44.250 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T21:17:44.250 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-26 21:17:21.637735196 +0000 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-26 21:11:25.188670659 +0000 2024-03-26T21:17:44.311 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-26 21:11:25.188670659 +0000 2024-03-26T21:17:44.312 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-26 21:11:25.188670659 +0000 2024-03-26T21:17:44.312 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T21:17:44.378 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-26T21:17:44.378 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-26T21:17:44.378 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000210137 s, 2.4 MB/s 2024-03-26T21:17:44.379 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T21:17:44.436 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-03-26T21:17:44.491 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T21:17:44.491 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:44.492 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T21:17:44.492 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:44.492 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:44.492 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-26 21:17:21.638735175 +0000 2024-03-26T21:17:44.492 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-26 21:11:25.532663440 +0000 2024-03-26T21:17:44.492 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-26 21:11:25.533663419 +0000 2024-03-26T21:17:44.492 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-26 21:11:25.532663440 +0000 2024-03-26T21:17:44.492 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T21:17:44.559 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-26T21:17:44.559 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-26T21:17:44.559 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000199141 s, 2.6 MB/s 2024-03-26T21:17:44.561 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T21:17:44.616 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-03-26T21:17:44.671 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T21:17:44.671 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:44.671 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T21:17:44.671 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:44.671 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:44.672 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-26 21:17:21.638735175 +0000 2024-03-26T21:17:44.672 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-26 21:11:25.889655947 +0000 2024-03-26T21:17:44.672 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-26 21:11:25.889655947 +0000 2024-03-26T21:17:44.672 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-26 21:11:25.889655947 +0000 2024-03-26T21:17:44.672 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T21:17:44.737 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-26T21:17:44.737 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-26T21:17:44.737 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000205776 s, 2.5 MB/s 2024-03-26T21:17:44.738 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T21:17:44.796 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-03-26T21:17:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T21:17:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:17:44.850 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-26T21:17:44.850 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:17:44.850 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:17:44.850 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-26 21:17:21.639735154 +0000 2024-03-26T21:17:44.851 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-26 21:11:26.218649042 +0000 2024-03-26T21:17:44.851 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-26 21:11:26.218649042 +0000 2024-03-26T21:17:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-26 21:11:26.218649042 +0000 2024-03-26T21:17:44.851 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T21:17:44.916 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-26T21:17:44.916 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-26T21:17:44.916 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000185361 s, 2.8 MB/s 2024-03-26T21:17:44.917 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T21:17:44.973 INFO:tasks.cephadm:Deploying osd.0 on smithi088 with /dev/vg_nvme/lv_4... 2024-03-26T21:17:44.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T21:17:45.206 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:17:45.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:45.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:45.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:45.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:45.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:45.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:45.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:17:45.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T21:17:45.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:17:45.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:45.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: Deploying daemon mgr.x on smithi138 2024-03-26T21:17:45.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[30075]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:45.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:45.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:45.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:45.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:45.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:45.988 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:45.988 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:17:45.988 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T21:17:45.988 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:17:45.988 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:45.988 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: Deploying daemon mgr.x on smithi138 2024-03-26T21:17:45.989 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:45 smithi088 ceph-mon[39202]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:46.106 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:46.107 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:46.107 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:46.107 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:46.108 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:46.108 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:46.108 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:17:46.108 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T21:17:46.108 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:17:46.108 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:46.109 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: Deploying daemon mgr.x on smithi138 2024-03-26T21:17:46.109 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:45 smithi138 ceph-mon[33706]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:46.975 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:46 smithi138 systemd[1]: Starting Ceph mgr.x for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:17:47.205 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T21:17:47.205 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T21:17:47.205 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-26T21:17:47.205 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-26T21:17:47.205 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110047 s, 953 MB/s 2024-03-26T21:17:47.205 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-26T21:17:47.353 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 podman[35779]: 2024-03-26 21:17:47.007175001 +0000 UTC m=+0.109898349 container create 216f1d336fb9eb530df0132417c2156d2ad42cb4bb40bcef35c65080022c15a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:17:47.353 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 podman[35779]: 2024-03-26 21:17:46.914771114 +0000 UTC m=+0.017494490 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:17:47.353 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 podman[35779]: 2024-03-26 21:17:47.251180239 +0000 UTC m=+0.353903596 container init 216f1d336fb9eb530df0132417c2156d2ad42cb4bb40bcef35c65080022c15a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T21:17:47.353 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 podman[35779]: 2024-03-26 21:17:47.256556487 +0000 UTC m=+0.359279837 container start 216f1d336fb9eb530df0132417c2156d2ad42cb4bb40bcef35c65080022c15a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T21:17:47.353 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 bash[35779]: 216f1d336fb9eb530df0132417c2156d2ad42cb4bb40bcef35c65080022c15a7 2024-03-26T21:17:47.724 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 systemd[1]: Started Ceph mgr.x for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:17:47.852 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_4 2024-03-26T21:17:48.087 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:17:48.124 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:47.817+0000 7f2b8c18c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:17:48.124 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:47 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:47.913+0000 7f2b8c18c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:17:48.380 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-mon[33706]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:48.380 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.380 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.380 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.380 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.380 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:48.453 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[30075]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:48.453 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.453 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.453 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.453 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.453 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:48.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[39202]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:48.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:48.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:48.682 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:48.595+0000 7f2b8c18c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:17:48.937 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:48.688+0000 7f2b8c18c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:17:48.937 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:48.845+0000 7f2b8c18c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:17:48.937 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:48.943+0000 7f2b8c18c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:17:49.224 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:49.026+0000 7f2b8c18c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:17:49.571 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:49.290+0000 7f2b8c18c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:17:49.571 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:49.398+0000 7f2b8c18c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:17:49.571 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:49.482+0000 7f2b8c18c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:17:49.571 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:49.577+0000 7f2b8c18c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:17:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.686+0000 7f1022f43700 1 -- 172.21.15.88:0/67284649 >> v1:172.21.15.88:6790/0 conn(0x7f101c102970 legacy=0x7f101c102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.686+0000 7f1022f43700 1 -- 172.21.15.88:0/67284649 shutdown_connections 2024-03-26T21:17:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.686+0000 7f1022f43700 1 -- 172.21.15.88:0/67284649 >> 172.21.15.88:0/67284649 conn(0x7f101c0fc0b0 msgr2=0x7f101c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:17:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.686+0000 7f1022f43700 1 -- 172.21.15.88:0/67284649 shutdown_connections 2024-03-26T21:17:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.686+0000 7f1022f43700 1 -- 172.21.15.88:0/67284649 wait complete. 2024-03-26T21:17:49.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1022f43700 1 Processor -- start 2024-03-26T21:17:49.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1022f43700 1 -- start start 2024-03-26T21:17:49.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1022f43700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f101c1013d0 con 0x7f101c1094c0 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1022f43700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f101c1a3910 con 0x7f101c105860 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1022f43700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f101c1a4b30 con 0x7f101c102970 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1020cdf700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f101c102970 0x7f101c19a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:39564/0 (socket says 172.21.15.88:39564) 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1020cdf700 1 -- 172.21.15.88:0/280932528 learned_addr learned my addr 172.21.15.88:0/280932528 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474829701 0 0) 0x7f101c1a4b30 con 0x7f101c102970 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f100c0050b0 con 0x7f101c102970 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107745542 0 0) 0x7f101c1013d0 con 0x7f101c1094c0 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f101c1a4b30 con 0x7f101c1094c0 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744160622 0 0) 0x7f101c1a3910 con 0x7f101c105860 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.687+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f101c1013d0 con 0x7f101c105860 2024-03-26T21:17:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2336461369 0 0) 0x7f100c0050b0 con 0x7f101c102970 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f101c1a3910 con 0x7f101c102970 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f1010004480 con 0x7f101c102970 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 398643674 0 0) 0x7f101c1a4b30 con 0x7f101c1094c0 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f100c0050b0 con 0x7f101c1094c0 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f1014001c10 con 0x7f101c1094c0 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3084191388 0 0) 0x7f101c1013d0 con 0x7f101c105860 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f101c1a4b30 con 0x7f101c105860 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f1008001830 con 0x7f101c105860 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3849368299 0 0) 0x7f101c1a3910 con 0x7f101c102970 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 >> v1:172.21.15.138:6789/0 conn(0x7f101c105860 legacy=0x7f101c19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 >> v1:172.21.15.88:6789/0 conn(0x7f101c1094c0 legacy=0x7f101c1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c1a5d70 con 0x7f101c102970 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f101c1a2960 con 0x7f101c102970 2024-03-26T21:17:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f1010001a30 con 0x7f101c102970 2024-03-26T21:17:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f10100050d0 con 0x7f101c102970 2024-03-26T21:17:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.688+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f101c1a2ea0 con 0x7f101c102970 2024-03-26T21:17:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.689+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 13) ==== 46487+0+0 (unknown 1525748001 0 0) 0x7f1010005350 con 0x7f101c102970 2024-03-26T21:17:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.689+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 396798714 0 0) 0x7f1010047b90 con 0x7f101c102970 2024-03-26T21:17:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.689+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c068f30 con 0x7f101c102970 2024-03-26T21:17:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.695+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f1010017500 con 0x7f101c102970 2024-03-26T21:17:49.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:49.911+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f101c1a3200 con 0x7f100c03a7d0 2024-03-26T21:17:49.974 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:49.753+0000 7f2b8c18c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: Reconfiguring daemon mgr.y on smithi088 2024-03-26T21:17:49.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:49 smithi138 ceph-mon[33706]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:17:50.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:17:50.055 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:50.055 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: Reconfiguring daemon mgr.y on smithi088 2024-03-26T21:17:50.055 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[30075]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:17:50.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:50.060 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: Reconfiguring daemon mgr.y on smithi088 2024-03-26T21:17:50.060 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:49 smithi088 ceph-mon[39202]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:50 smithi138 ceph-mon[33706]: from='client.34115 v1:172.21.15.88:0/280932528' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:17:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:17:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:50.975 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:50.721+0000 7f2b8c18c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:17:50.975 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:50.853+0000 7f2b8c18c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:17:51.126 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[30075]: from='client.34115 v1:172.21.15.88:0/280932528' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:51.126 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:17:51.126 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:17:51.126 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:51.126 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[39202]: from='client.34115 v1:172.21.15.88:0/280932528' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:17:51.126 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:17:51.127 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:17:51.127 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:51.465 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:51.187+0000 7f2b8c18c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:17:51.465 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:51.276+0000 7f2b8c18c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:17:51.465 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:51.470+0000 7f2b8c18c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:17:51.724 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:51.656+0000 7f2b8c18c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:17:51.821 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[39202]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:51.822 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:51.823 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:51.823 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[30075]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:51.823 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:51.823 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:51.823 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:51.943+0000 7f2b8c18c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:52.205+0000 7f2b8c18c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-mon[33706]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:17:52.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:52.474 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:52.440+0000 7f2b8c18c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:17:53.474 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:53.239+0000 7f2b8c18c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:17:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2159740475' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]: dispatch 2024-03-26T21:17:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-mon[33706]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]: dispatch 2024-03-26T21:17:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-mon[33706]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]': finished 2024-03-26T21:17:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-mon[33706]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T21:17:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:17:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2159740475' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]: dispatch 2024-03-26T21:17:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[30075]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]: dispatch 2024-03-26T21:17:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[30075]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]': finished 2024-03-26T21:17:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[30075]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T21:17:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:17:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:17:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2159740475' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]: dispatch 2024-03-26T21:17:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[39202]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]: dispatch 2024-03-26T21:17:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[39202]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09199105-49c9-4d89-8c77-9193d4a78bf0"}]': finished 2024-03-26T21:17:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[39202]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T21:17:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:53 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:17:54.114 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:53.808+0000 7f2b8c18c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:17:54.114 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:53 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:53.904+0000 7f2b8c18c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:17:54.114 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:17:54 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:17:54.119+0000 7f2b8c18c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:17:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:54 smithi138 ceph-mon[33706]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4245412330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:17:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:54 smithi088 ceph-mon[39202]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4245412330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:17:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:54 smithi088 ceph-mon[30075]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4245412330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:17:55.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:17:55.142+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mgrmap(e 14) ==== 92383+0+0 (unknown 3897324969 0 0) 0x7f10100134d0 con 0x7f101c102970 2024-03-26T21:17:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:55 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:17:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:55 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:17:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:55 smithi138 ceph-mon[33706]: Standby manager daemon x started 2024-03-26T21:17:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:55 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:17:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:55 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:17:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:17:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:17:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[30075]: Standby manager daemon x started 2024-03-26T21:17:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:17:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:17:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:17:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:17:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[39202]: Standby manager daemon x started 2024-03-26T21:17:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:17:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:55 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/1128703862' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:17:56.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:56 smithi138 ceph-mon[33706]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:56.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:56 smithi138 ceph-mon[33706]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-26T21:17:56.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:56 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:17:56.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:56 smithi088 ceph-mon[30075]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:56.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:56 smithi088 ceph-mon[30075]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-26T21:17:56.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:56 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:17:56.406 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:56 smithi088 ceph-mon[39202]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:56.406 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:56 smithi088 ceph-mon[39202]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-26T21:17:56.406 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:56 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:17:58.342 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:17:58 smithi088 ceph-mon[30075]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:58.343 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:17:58 smithi088 ceph-mon[39202]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:17:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:17:58 smithi138 ceph-mon[33706]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:00.345 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:00 smithi088 ceph-mon[30075]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:00.345 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:00 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T21:18:00.345 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:00 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:00.346 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:00 smithi088 ceph-mon[39202]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:00.346 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:00 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T21:18:00.346 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:00 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:00 smithi138 ceph-mon[33706]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:00 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T21:18:00.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:00 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:01.402 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:01 smithi088 ceph-mon[39202]: Deploying daemon osd.0 on smithi088 2024-03-26T21:18:01.402 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:01 smithi088 ceph-mon[30075]: Deploying daemon osd.0 on smithi088 2024-03-26T21:18:01.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:01 smithi138 ceph-mon[33706]: Deploying daemon osd.0 on smithi088 2024-03-26T21:18:02.321 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:02 smithi088 ceph-mon[39202]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:02.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:02 smithi088 ceph-mon[30075]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:02 smithi138 ceph-mon[33706]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:04.338 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:04 smithi088 ceph-mon[30075]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:04.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:04 smithi088 ceph-mon[39202]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:04.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:04 smithi138 ceph-mon[33706]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:05 smithi138 ceph-mon[33706]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[30075]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:05 smithi088 ceph-mon[39202]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[30075]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:08.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:08.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:08.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[30075]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T21:18:08.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[39202]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:08.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:08.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:08.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:08 smithi088 ceph-mon[39202]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T21:18:08.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:08 smithi138 ceph-mon[33706]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:08 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:08 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:08 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:08 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:08 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:08 smithi138 ceph-mon[33706]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T21:18:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:08.613+0000 7f1019ffb700 1 -- 172.21.15.88:0/280932528 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3531913810) 0x7f101c1a3200 con 0x7f100c03a7d0 2024-03-26T21:18:08.608 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 0 on host 'smithi088' 2024-03-26T21:18:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:08.616+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 >> v1:172.21.15.88:6800/4097082509 conn(0x7f100c03a7d0 legacy=0x7f100c03cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:08.616+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 >> v1:172.21.15.88:6790/0 conn(0x7f101c102970 legacy=0x7f101c19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:08.616+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 shutdown_connections 2024-03-26T21:18:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:08.616+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 >> 172.21.15.88:0/280932528 conn(0x7f101c0fc0b0 msgr2=0x7f101c0fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:18:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:08.616+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 shutdown_connections 2024-03-26T21:18:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:08.616+0000 7f1022f43700 1 -- 172.21.15.88:0/280932528 wait complete. 2024-03-26T21:18:09.254 DEBUG:teuthology.orchestra.run.smithi088:osd.0> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T21:18:09.256 INFO:tasks.cephadm:Deploying osd.1 on smithi088 with /dev/vg_nvme/lv_3... 2024-03-26T21:18:09.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[30075]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:09.815 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:09.816 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:09.816 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T21:18:09.816 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T21:18:09.816 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:09.816 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:09.816 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:09.816 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-mon[39202]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:09.817 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0[43605]: 2024-03-26T21:18:09.603+0000 7f0ef1c49700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T21:18:09.817 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:18:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0[43605]: 2024-03-26T21:18:09.605+0000 7f0eeda3c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:18:09.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:09.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:09.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T21:18:09.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T21:18:09.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:09.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:09.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:09.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:09 smithi138 ceph-mon[33706]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:10.006 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:18:10.653 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[30075]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:18:10.653 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[30075]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T21:18:10.653 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:10.653 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:10.653 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[39202]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:18:10.654 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[39202]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T21:18:10.654 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:10.654 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:10 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:10 smithi138 ceph-mon[33706]: from='osd.0 v1:172.21.15.88:6801/3845975585' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:18:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:10 smithi138 ceph-mon[33706]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T21:18:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:10 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:10 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:11.811 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:18:11.811 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[30075]: osd.0 v1:172.21.15.88:6801/3845975585 boot 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[30075]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[30075]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[39202]: osd.0 v1:172.21.15.88:6801/3845975585 boot 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[39202]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:11.812 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:11 smithi088 ceph-mon[39202]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:11 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:18:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:11 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:18:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:11 smithi138 ceph-mon[33706]: osd.0 v1:172.21.15.88:6801/3845975585 boot 2024-03-26T21:18:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:11 smithi138 ceph-mon[33706]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T21:18:11.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:11 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:18:11.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:11 smithi138 ceph-mon[33706]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:18:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T21:18:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T21:18:12.501 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-26T21:18:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-26T21:18:12.501 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125811 s, 833 MB/s 2024-03-26T21:18:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-26T21:18:12.843 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:12 smithi088 ceph-mon[39202]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T21:18:12.843 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:12 smithi088 ceph-mon[30075]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T21:18:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:12 smithi138 ceph-mon[33706]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T21:18:13.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_3 2024-03-26T21:18:13.797 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:13 smithi088 ceph-mon[30075]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:13.797 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:13 smithi088 ceph-mon[39202]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:13.935 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:18:13.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:13 smithi138 ceph-mon[33706]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi088 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi088 to 16064M 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi088 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi088 to 16064M 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi088 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi088 to 16064M 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:15.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:15.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.441+0000 7f7892cd8700 1 -- 172.21.15.88:0/4007658958 >> v1:172.21.15.88:6789/0 conn(0x7f788c106880 legacy=0x7f788c108cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.441+0000 7f7892cd8700 1 -- 172.21.15.88:0/4007658958 shutdown_connections 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.441+0000 7f7892cd8700 1 -- 172.21.15.88:0/4007658958 >> 172.21.15.88:0/4007658958 conn(0x7f788c0fd9f0 msgr2=0x7f788c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.441+0000 7f7892cd8700 1 -- 172.21.15.88:0/4007658958 shutdown_connections 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.441+0000 7f7892cd8700 1 -- 172.21.15.88:0/4007658958 wait complete. 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.441+0000 7f7892cd8700 1 Processor -- start 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.441+0000 7f7892cd8700 1 -- start start 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7892cd8700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f788c1a4b10 con 0x7f788c106880 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7892cd8700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f788c1a5d50 con 0x7f788c102450 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7892cd8700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f788c1a6f90 con 0x7f788c10a5c0 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7883fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f788c106880 0x7f788c19f8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47638/0 (socket says 172.21.15.88:47638) 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7883fff700 1 -- 172.21.15.88:0/2729702218 learned_addr learned my addr 172.21.15.88:0/2729702218 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676334331 0 0) 0x7f788c1a6f90 con 0x7f788c10a5c0 2024-03-26T21:18:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78740050b0 con 0x7f788c10a5c0 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550998906 0 0) 0x7f788c1a4b10 con 0x7f788c106880 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f788c1a6f90 con 0x7f788c106880 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 862985124 0 0) 0x7f788c1a5d50 con 0x7f788c102450 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f788c1a4b10 con 0x7f788c102450 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2353130787 0 0) 0x7f78740050b0 con 0x7f788c10a5c0 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f788c1a5d50 con 0x7f788c10a5c0 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7884001590 con 0x7f788c10a5c0 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1291996874 0 0) 0x7f788c1a6f90 con 0x7f788c106880 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78740050b0 con 0x7f788c106880 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f787c004510 con 0x7f788c106880 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3542926553 0 0) 0x7f788c1a4b10 con 0x7f788c102450 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f788c1a6f90 con 0x7f788c102450 2024-03-26T21:18:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7888001830 con 0x7f788c102450 2024-03-26T21:18:15.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3221005268 0 0) 0x7f788c1a5d50 con 0x7f788c10a5c0 2024-03-26T21:18:15.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 >> v1:172.21.15.138:6789/0 conn(0x7f788c102450 legacy=0x7f788c076300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:15.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 >> v1:172.21.15.88:6789/0 conn(0x7f788c106880 legacy=0x7f788c19f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:15.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788c1a81d0 con 0x7f788c10a5c0 2024-03-26T21:18:15.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.442+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f788c1a4d40 con 0x7f788c10a5c0 2024-03-26T21:18:15.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.443+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f7884003e90 con 0x7f788c10a5c0 2024-03-26T21:18:15.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.443+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7884004d40 con 0x7f788c10a5c0 2024-03-26T21:18:15.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.443+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f788c1a5280 con 0x7f788c10a5c0 2024-03-26T21:18:15.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.444+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f788c068f30 con 0x7f788c10a5c0 2024-03-26T21:18:15.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.444+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 14) ==== 92383+0+0 (unknown 3897324969 0 0) 0x7f7884003a00 con 0x7f788c10a5c0 2024-03-26T21:18:15.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.444+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(9..9 src has 1..9) ==== 1541+0+0 (unknown 4151208767 0 0) 0x7f7884087940 con 0x7f788c10a5c0 2024-03-26T21:18:15.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.449+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f7884056140 con 0x7f788c10a5c0 2024-03-26T21:18:15.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:15.657+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f788c1a5610 con 0x7f787406f470 2024-03-26T21:18:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:15 smithi138 ceph-mon[33706]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:15 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:18:16.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:15 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:18:16.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:15 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[39202]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[30075]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:18:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:15 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:16 smithi138 ceph-mon[33706]: from='client.34142 v1:172.21.15.88:0/2729702218' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:18:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:16 smithi088 ceph-mon[30075]: from='client.34142 v1:172.21.15.88:0/2729702218' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:18:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:16 smithi088 ceph-mon[39202]: from='client.34142 v1:172.21.15.88:0/2729702218' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:18:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:17 smithi138 ceph-mon[33706]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:17 smithi088 ceph-mon[30075]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:17 smithi088 ceph-mon[39202]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:19.175 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/278922788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]: dispatch 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[39202]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]: dispatch 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[39202]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]': finished 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[39202]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/278922788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]: dispatch 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[30075]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]: dispatch 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[30075]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]': finished 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[30075]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T21:18:19.176 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/278922788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]: dispatch 2024-03-26T21:18:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:18 smithi138 ceph-mon[33706]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]: dispatch 2024-03-26T21:18:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:18 smithi138 ceph-mon[33706]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2759e98-1df5-4e4d-a7f0-719202f0bc5b"}]': finished 2024-03-26T21:18:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:18 smithi138 ceph-mon[33706]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T21:18:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:18 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:19 smithi138 ceph-mon[33706]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1033893810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:18:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:19 smithi088 ceph-mon[30075]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1033893810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:18:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:19 smithi088 ceph-mon[39202]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1033893810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:18:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:22 smithi138 ceph-mon[33706]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:22.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:22 smithi088 ceph-mon[30075]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:22 smithi088 ceph-mon[39202]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:24 smithi138 ceph-mon[33706]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:24 smithi088 ceph-mon[30075]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:24 smithi088 ceph-mon[39202]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T21:18:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T21:18:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T21:18:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:26 smithi138 ceph-mon[33706]: Deploying daemon osd.1 on smithi088 2024-03-26T21:18:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:26 smithi138 ceph-mon[33706]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:26 smithi088 ceph-mon[30075]: Deploying daemon osd.1 on smithi088 2024-03-26T21:18:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:26 smithi088 ceph-mon[30075]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:26 smithi088 ceph-mon[39202]: Deploying daemon osd.1 on smithi088 2024-03-26T21:18:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:26 smithi088 ceph-mon[39202]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:28 smithi138 ceph-mon[33706]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:28 smithi088 ceph-mon[30075]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:28 smithi088 ceph-mon[39202]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:30.396 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:30 smithi088 ceph-mon[30075]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:30.396 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:30 smithi088 ceph-mon[39202]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:30 smithi138 ceph-mon[33706]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:31 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:31 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:31.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:31 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:31.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:31 smithi138 ceph-mon[33706]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[30075]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:31 smithi088 ceph-mon[39202]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:33.903 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 1 on host 'smithi088' 2024-03-26T21:18:33.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:33.908+0000 7f7881ffb700 1 -- 172.21.15.88:0/2729702218 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 2391889126) 0x7f788c1a5610 con 0x7f787406f470 2024-03-26T21:18:33.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:33.911+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 >> v1:172.21.15.88:6800/4097082509 conn(0x7f787406f470 legacy=0x7f7874071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:33.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:33.911+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 >> v1:172.21.15.88:6790/0 conn(0x7f788c10a5c0 legacy=0x7f788c1a3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:33.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:33.911+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 shutdown_connections 2024-03-26T21:18:33.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:33.911+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 >> 172.21.15.88:0/2729702218 conn(0x7f788c0fd9f0 msgr2=0x7f788c109400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:18:33.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:33.911+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 shutdown_connections 2024-03-26T21:18:33.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:33.911+0000 7f7892cd8700 1 -- 172.21.15.88:0/2729702218 wait complete. 2024-03-26T21:18:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:34 smithi138 ceph-mon[33706]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:34 smithi138 ceph-mon[33706]: from='osd.1 v1:172.21.15.88:6805/698625447' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:18:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:34 smithi138 ceph-mon[33706]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:18:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:34 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:34 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[39202]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[39202]: from='osd.1 v1:172.21.15.88:6805/698625447' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[39202]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[30075]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[30075]: from='osd.1 v1:172.21.15.88:6805/698625447' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[30075]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:34 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:34.526 DEBUG:teuthology.orchestra.run.smithi088:osd.1> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T21:18:34.528 INFO:tasks.cephadm:Deploying osd.2 on smithi088 with /dev/vg_nvme/lv_2... 2024-03-26T21:18:34.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T21:18:34.813 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[30075]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[30075]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[30075]: from='osd.1 v1:172.21.15.88:6805/698625447' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[30075]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[39202]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T21:18:35.211 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[39202]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T21:18:35.212 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:35.212 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[39202]: from='osd.1 v1:172.21.15.88:6805/698625447' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:35.212 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[39202]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:35.212 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:35.212 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:35 smithi138 ceph-mon[33706]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T21:18:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:35 smithi138 ceph-mon[33706]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T21:18:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:35 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:35 smithi138 ceph-mon[33706]: from='osd.1 v1:172.21.15.88:6805/698625447' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:35 smithi138 ceph-mon[33706]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:18:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:35 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:35 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:35.930 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1[49547]: 2024-03-26T21:18:35.570+0000 7f42c6f54700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T21:18:35.930 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:18:35 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1[49547]: 2024-03-26T21:18:35.573+0000 7f42be53e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:18:36.213 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[30075]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:36.213 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[30075]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:18:36.213 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[30075]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T21:18:36.213 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:36.213 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:36.213 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[30075]: from='osd.1 ' entity='osd.1' 2024-03-26T21:18:36.214 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[39202]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:36.214 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[39202]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:18:36.214 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[39202]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T21:18:36.214 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:36.214 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:36.214 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:36 smithi088 ceph-mon[39202]: from='osd.1 ' entity='osd.1' 2024-03-26T21:18:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:36 smithi138 ceph-mon[33706]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:36 smithi138 ceph-mon[33706]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:18:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:36 smithi138 ceph-mon[33706]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T21:18:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:36 smithi138 ceph-mon[33706]: from='osd.1 ' entity='osd.1' 2024-03-26T21:18:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:18:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:18:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[30075]: osd.1 v1:172.21.15.88:6805/698625447 boot 2024-03-26T21:18:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[30075]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T21:18:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:18:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:18:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[39202]: osd.1 v1:172.21.15.88:6805/698625447 boot 2024-03-26T21:18:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[39202]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T21:18:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:37.399 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T21:18:37.399 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T21:18:37.399 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-26T21:18:37.399 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-26T21:18:37.399 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126051 s, 832 MB/s 2024-03-26T21:18:37.399 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-26T21:18:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:37 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:18:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:37 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:18:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:37 smithi138 ceph-mon[33706]: osd.1 v1:172.21.15.88:6805/698625447 boot 2024-03-26T21:18:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:37 smithi138 ceph-mon[33706]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T21:18:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:18:38.069 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_2 2024-03-26T21:18:38.350 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:38 smithi088 ceph-mon[39202]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:38.351 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:38 smithi088 ceph-mon[30075]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:38 smithi138 ceph-mon[33706]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:18:38.623 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:18:39.345 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:39 smithi088 ceph-mon[30075]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T21:18:39.345 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:39 smithi088 ceph-mon[39202]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T21:18:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:39 smithi138 ceph-mon[33706]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T21:18:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:40.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:40.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:18:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:40.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.964+0000 7f517905c700 1 -- 172.21.15.88:0/3558301441 >> v1:172.21.15.88:6790/0 conn(0x7f51741094a0 legacy=0x7f517410b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:40.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.964+0000 7f517905c700 1 -- 172.21.15.88:0/3558301441 shutdown_connections 2024-03-26T21:18:40.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.964+0000 7f517905c700 1 -- 172.21.15.88:0/3558301441 >> 172.21.15.88:0/3558301441 conn(0x7f51740fc8d0 msgr2=0x7f51740fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:18:40.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.964+0000 7f517905c700 1 -- 172.21.15.88:0/3558301441 shutdown_connections 2024-03-26T21:18:40.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.964+0000 7f517905c700 1 -- 172.21.15.88:0/3558301441 wait complete. 2024-03-26T21:18:40.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.964+0000 7f517905c700 1 Processor -- start 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f517905c700 1 -- start start 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f517905c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f517406ae80 con 0x7f5174105760 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f517905c700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f51741a3b10 con 0x7f5174101330 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f517905c700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f51741a4d30 con 0x7f51741094a0 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f517259c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5174105760 0x7f517419dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37864/0 (socket says 172.21.15.88:37864) 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f517259c700 1 -- 172.21.15.88:0/1083660544 learned_addr learned my addr 172.21.15.88:0/1083660544 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f517359e700 1 --1- 172.21.15.88:0/1083660544 >> v1:172.21.15.88:6790/0 conn(0x7f51741094a0 0x7f51741a13e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:44434/0 (socket says 172.21.15.88:44434) 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093587785 0 0) 0x7f517406ae80 con 0x7f5174105760 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f515c0035b0 con 0x7f5174105760 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 38376357 0 0) 0x7f51741a4d30 con 0x7f51741094a0 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f517406ae80 con 0x7f51741094a0 2024-03-26T21:18:40.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3105929797 0 0) 0x7f51741a3b10 con 0x7f5174101330 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.965+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51741a4d30 con 0x7f5174101330 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3908263 0 0) 0x7f515c0035b0 con 0x7f5174105760 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51741a3b10 con 0x7f5174105760 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5164001930 con 0x7f5174105760 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2689996469 0 0) 0x7f51741a4d30 con 0x7f5174101330 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f515c0035b0 con 0x7f5174101330 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f51680018c0 con 0x7f5174101330 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1759405089 0 0) 0x7f517406ae80 con 0x7f51741094a0 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51741a4d30 con 0x7f51741094a0 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f516c0047b0 con 0x7f51741094a0 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 5074804 0 0) 0x7f51741a3b10 con 0x7f5174105760 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 >> v1:172.21.15.88:6790/0 conn(0x7f51741094a0 legacy=0x7f51741a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 >> v1:172.21.15.138:6789/0 conn(0x7f5174101330 legacy=0x7f517419a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51741a5f70 con 0x7f5174105760 2024-03-26T21:18:40.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f51741a4f60 con 0x7f5174105760 2024-03-26T21:18:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f51741a54a0 con 0x7f5174105760 2024-03-26T21:18:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f5164001d00 con 0x7f5174105760 2024-03-26T21:18:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.966+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5164005e00 con 0x7f5174105760 2024-03-26T21:18:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.967+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5160005120 con 0x7f5174105760 2024-03-26T21:18:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.968+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 14) ==== 92383+0+0 (unknown 3897324969 0 0) 0x7f5164007090 con 0x7f5174105760 2024-03-26T21:18:40.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.968+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(14..14 src has 1..14) ==== 1833+0+0 (unknown 869136723 0 0) 0x7f5164088a00 con 0x7f5174105760 2024-03-26T21:18:40.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:40.973+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f5164057000 con 0x7f5174105760 2024-03-26T21:18:41.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:41.190+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5160000b90 con 0x7f515c0739b0 2024-03-26T21:18:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:41 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi088 2024-03-26T21:18:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:41 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi088 to 8032M 2024-03-26T21:18:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:41 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi088 2024-03-26T21:18:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:41 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi088 to 8032M 2024-03-26T21:18:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:41 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi088 2024-03-26T21:18:41.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:41 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi088 to 8032M 2024-03-26T21:18:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:42 smithi138 ceph-mon[33706]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:42 smithi138 ceph-mon[33706]: from='client.14271 v1:172.21.15.88:0/1083660544' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:18:42.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:18:42.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:18:42.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[30075]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[30075]: from='client.14271 v1:172.21.15.88:0/1083660544' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[39202]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[39202]: from='client.14271 v1:172.21.15.88:0/1083660544' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:18:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:44.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:44 smithi138 ceph-mon[33706]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:44.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/111978708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bb8665c-3450-49dc-86fb-0d086ffb8ec1"}]: dispatch 2024-03-26T21:18:44.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/111978708' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bb8665c-3450-49dc-86fb-0d086ffb8ec1"}]': finished 2024-03-26T21:18:44.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:44 smithi138 ceph-mon[33706]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T21:18:44.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:44 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:18:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[39202]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/111978708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bb8665c-3450-49dc-86fb-0d086ffb8ec1"}]: dispatch 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/111978708' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bb8665c-3450-49dc-86fb-0d086ffb8ec1"}]': finished 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[39202]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[30075]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/111978708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bb8665c-3450-49dc-86fb-0d086ffb8ec1"}]: dispatch 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/111978708' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bb8665c-3450-49dc-86fb-0d086ffb8ec1"}]': finished 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[30075]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T21:18:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:44 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:18:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313109311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:18:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313109311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:18:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313109311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:18:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:46 smithi138 ceph-mon[33706]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:46 smithi088 ceph-mon[30075]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:46 smithi088 ceph-mon[39202]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:48.277 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:48 smithi088 ceph-mon[30075]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:48.277 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:48 smithi088 ceph-mon[39202]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:48 smithi138 ceph-mon[33706]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:50.354 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:50 smithi088 ceph-mon[30075]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:50.354 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:50 smithi088 ceph-mon[39202]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:50 smithi138 ceph-mon[33706]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:51.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T21:18:51.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:51.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T21:18:51.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T21:18:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:18:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:52 smithi138 ceph-mon[33706]: Deploying daemon osd.2 on smithi088 2024-03-26T21:18:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:52 smithi138 ceph-mon[33706]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:52.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:52 smithi088 ceph-mon[39202]: Deploying daemon osd.2 on smithi088 2024-03-26T21:18:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:52 smithi088 ceph-mon[39202]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:52 smithi088 ceph-mon[30075]: Deploying daemon osd.2 on smithi088 2024-03-26T21:18:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:52 smithi088 ceph-mon[30075]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:54.460 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:54 smithi088 ceph-mon[30075]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:54.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:54 smithi088 ceph-mon[39202]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:54 smithi138 ceph-mon[33706]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:56 smithi138 ceph-mon[33706]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:56 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:56.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:56 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:56.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:56 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[30075]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:56.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[39202]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:56.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:56.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:18:56.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:56 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:18:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:18:58 smithi088 ceph-mon[30075]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:18:58 smithi088 ceph-mon[39202]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:18:58 smithi138 ceph-mon[33706]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:18:59.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:59.943+0000 7f515bfff700 1 -- 172.21.15.88:0/1083660544 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 1789880122) 0x7f5160000b90 con 0x7f515c0739b0 2024-03-26T21:18:59.938 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 2 on host 'smithi088' 2024-03-26T21:18:59.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:59.946+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 >> v1:172.21.15.88:6800/4097082509 conn(0x7f515c0739b0 legacy=0x7f515c075e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:59.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:59.946+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 >> v1:172.21.15.88:6789/0 conn(0x7f5174105760 legacy=0x7f517419dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:18:59.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:59.946+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 shutdown_connections 2024-03-26T21:18:59.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:59.946+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 >> 172.21.15.88:0/1083660544 conn(0x7f51740fc8d0 msgr2=0x7f517410c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:18:59.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:59.946+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 shutdown_connections 2024-03-26T21:18:59.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:18:59.946+0000 7f517905c700 1 -- 172.21.15.88:0/1083660544 wait complete. 2024-03-26T21:19:00.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[30075]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:00.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[30075]: from='osd.2 v1:172.21.15.88:6809/2517903952' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[30075]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[39202]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[39202]: from='osd.2 v1:172.21.15.88:6809/2517903952' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[39202]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:00 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:00 smithi138 ceph-mon[33706]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:00 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:00 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:00 smithi138 ceph-mon[33706]: from='osd.2 v1:172.21.15.88:6809/2517903952' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:19:00.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:00 smithi138 ceph-mon[33706]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:19:00.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:00 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:00 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:00.531 DEBUG:teuthology.orchestra.run.smithi088:osd.2> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T21:19:00.533 INFO:tasks.cephadm:Deploying osd.3 on smithi088 with /dev/vg_nvme/lv_1... 2024-03-26T21:19:00.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T21:19:00.818 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:19:01.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:01 smithi138 ceph-mon[33706]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T21:19:01.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:01 smithi138 ceph-mon[33706]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T21:19:01.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:01 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:01.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:01 smithi138 ceph-mon[33706]: from='osd.2 v1:172.21.15.88:6809/2517903952' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:01.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:01 smithi138 ceph-mon[33706]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[30075]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[30075]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[30075]: from='osd.2 v1:172.21.15.88:6809/2517903952' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[30075]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[39202]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[39202]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[39202]: from='osd.2 v1:172.21.15.88:6809/2517903952' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:01.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-mon[39202]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:01.485 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2[56021]: 2024-03-26T21:19:01.222+0000 7f0c35cf6700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T21:19:01.485 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:01 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2[56021]: 2024-03-26T21:19:01.225+0000 7f0c31ae9700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:19:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:02 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:19:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:02 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:19:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:02 smithi138 ceph-mon[33706]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:02 smithi138 ceph-mon[33706]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:19:02.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:02 smithi138 ceph-mon[33706]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T21:19:02.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:02 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:02.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:02 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:02.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:19:02.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:19:02.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[30075]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:02.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[30075]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:19:02.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[30075]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T21:19:02.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:02.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:02.480 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:19:02.480 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:19:02.480 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[39202]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:02.480 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[39202]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:19:02.480 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[39202]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T21:19:02.480 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:02.480 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:02 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:03.264 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[39202]: osd.2 v1:172.21.15.88:6809/2517903952 boot 2024-03-26T21:19:03.264 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[39202]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T21:19:03.264 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:03.264 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:19:03.264 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[30075]: osd.2 v1:172.21.15.88:6809/2517903952 boot 2024-03-26T21:19:03.264 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[30075]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T21:19:03.264 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:03.265 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:03 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:19:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T21:19:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T21:19:03.482 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-26T21:19:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-26T21:19:03.482 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146236 s, 717 MB/s 2024-03-26T21:19:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-26T21:19:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:03 smithi138 ceph-mon[33706]: osd.2 v1:172.21.15.88:6809/2517903952 boot 2024-03-26T21:19:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:03 smithi138 ceph-mon[33706]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T21:19:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:03 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:19:03.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:03 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:19:04.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_1 2024-03-26T21:19:04.425 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[39202]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:04.425 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T21:19:04.425 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[39202]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T21:19:04.425 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57696]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57696]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57696]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57696]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57699]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57699]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57699]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57699]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57702]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57702]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.426 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57702]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.427 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57702]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.427 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57717]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:19:04.427 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57717]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.427 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57717]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.427 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57717]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.427 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[30075]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:04.427 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T21:19:04.427 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[30075]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T21:19:04.427 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:19:04.505 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 ceph-mon[33706]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:19:04.505 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T21:19:04.505 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 ceph-mon[33706]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T21:19:04.505 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57720]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57720]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57720]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57720]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57723]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.707 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57723]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57726]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57726]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57726]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57726]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57745]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57745]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57745]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57745]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57766]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57766]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.708 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57766]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.709 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57766]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.709 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57769]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:19:04.709 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57769]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.709 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57769]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.709 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:04 smithi088 sudo[57769]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:19:04.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36476]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36476]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36476]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36476]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36479]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36479]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36479]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:04 smithi138 sudo[36479]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:19:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[39202]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:05 smithi088 ceph-mon[30075]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:19:05.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:05 smithi138 ceph-mon[33706]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:19:06.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:06.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:06.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:06.438 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:06.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:06.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:06.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:06.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.949+0000 7efd09962700 1 -- 172.21.15.88:0/984955347 >> v1:172.21.15.88:6789/0 conn(0x7efd04106880 legacy=0x7efd04108cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:06.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.949+0000 7efd09962700 1 -- 172.21.15.88:0/984955347 shutdown_connections 2024-03-26T21:19:06.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.949+0000 7efd09962700 1 -- 172.21.15.88:0/984955347 >> 172.21.15.88:0/984955347 conn(0x7efd040fd9f0 msgr2=0x7efd040ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:19:06.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.949+0000 7efd09962700 1 -- 172.21.15.88:0/984955347 shutdown_connections 2024-03-26T21:19:06.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.949+0000 7efd09962700 1 -- 172.21.15.88:0/984955347 wait complete. 2024-03-26T21:19:06.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.949+0000 7efd09962700 1 Processor -- start 2024-03-26T21:19:06.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.949+0000 7efd09962700 1 -- start start 2024-03-26T21:19:06.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd09962700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd041984e0 con 0x7efd04106880 2024-03-26T21:19:06.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd09962700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd041a3cd0 con 0x7efd0410a5c0 2024-03-26T21:19:06.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd09962700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd041a4eb0 con 0x7efd04102450 2024-03-26T21:19:06.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd02ffd700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7efd04102450 0x7efd04197990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47938/0 (socket says 172.21.15.88:47938) 2024-03-26T21:19:06.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd02ffd700 1 -- 172.21.15.88:0/3831161993 learned_addr learned my addr 172.21.15.88:0/3831161993 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efcfa7fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efd04106880 0x7efd0419de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57182/0 (socket says 172.21.15.88:57182) 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225775410 0 0) 0x7efd041984e0 con 0x7efd04106880 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcf00035b0 con 0x7efd04106880 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2689593910 0 0) 0x7efd041a4eb0 con 0x7efd04102450 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd041984e0 con 0x7efd04102450 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029173619 0 0) 0x7efd041a3cd0 con 0x7efd0410a5c0 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd041a4eb0 con 0x7efd0410a5c0 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3479743665 0 0) 0x7efd041984e0 con 0x7efd04102450 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd041a3cd0 con 0x7efd04102450 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2332888606 0 0) 0x7efcf00035b0 con 0x7efd04106880 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd041984e0 con 0x7efd04106880 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.950+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7efcf4001890 con 0x7efd04102450 2024-03-26T21:19:06.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7efcec0044a0 con 0x7efd04106880 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3502325662 0 0) 0x7efd041a4eb0 con 0x7efd0410a5c0 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcf00035b0 con 0x7efd0410a5c0 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7efcfc001b30 con 0x7efd0410a5c0 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2213051849 0 0) 0x7efd041a3cd0 con 0x7efd04102450 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 >> v1:172.21.15.138:6789/0 conn(0x7efd0410a5c0 legacy=0x7efd041a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 >> v1:172.21.15.88:6789/0 conn(0x7efd04106880 legacy=0x7efd0419de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd041a60d0 con 0x7efd04102450 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd041a50e0 con 0x7efd04102450 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efd041a5620 con 0x7efd04102450 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7efcf4003b10 con 0x7efd04102450 2024-03-26T21:19:06.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.951+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7efcf4005c00 con 0x7efd04102450 2024-03-26T21:19:06.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.952+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce4005150 con 0x7efd04102450 2024-03-26T21:19:06.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.952+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7efcf4003680 con 0x7efd04102450 2024-03-26T21:19:06.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.953+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 2511+0+0 (unknown 1762152615 0 0) 0x7efcf4088db0 con 0x7efd04102450 2024-03-26T21:19:06.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:06.958+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7efcf4057110 con 0x7efd04102450 2024-03-26T21:19:07.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:07.168+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7efce4000bc0 con 0x7efcf006f4f0 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi088 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi088 to 5354M 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[30075]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi088 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi088 to 5354M 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[39202]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T21:19:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:07.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:07.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:07 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:07 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi088 2024-03-26T21:19:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:07 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi088 to 5354M 2024-03-26T21:19:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:07 smithi138 ceph-mon[33706]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T21:19:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:07 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:07.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:07 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:07.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:07 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:08.479 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:08 smithi088 ceph-mon[39202]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:08.479 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:08 smithi088 ceph-mon[39202]: from='client.34205 v1:172.21.15.88:0/3831161993' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:08.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:08 smithi088 ceph-mon[30075]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:08.479 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:08 smithi088 ceph-mon[30075]: from='client.34205 v1:172.21.15.88:0/3831161993' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:08 smithi138 ceph-mon[33706]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:08 smithi138 ceph-mon[33706]: from='client.34205 v1:172.21.15.88:0/3831161993' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:10.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[39202]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:10.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79900154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]: dispatch 2024-03-26T21:19:10.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[39202]: from='client.34211 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]: dispatch 2024-03-26T21:19:10.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[39202]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]': finished 2024-03-26T21:19:10.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[39202]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T21:19:10.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:10.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[30075]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:10.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79900154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]: dispatch 2024-03-26T21:19:10.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[30075]: from='client.34211 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]: dispatch 2024-03-26T21:19:10.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[30075]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]': finished 2024-03-26T21:19:10.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[30075]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T21:19:10.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:10 smithi138 ceph-mon[33706]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79900154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]: dispatch 2024-03-26T21:19:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:10 smithi138 ceph-mon[33706]: from='client.34211 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]: dispatch 2024-03-26T21:19:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:10 smithi138 ceph-mon[33706]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a727a6dd-71c5-4867-a2ad-91423368c35c"}]': finished 2024-03-26T21:19:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:10 smithi138 ceph-mon[33706]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T21:19:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:10 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1185558213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:19:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:11 smithi138 ceph-mon[33706]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:11.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1185558213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:19:11.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:11 smithi088 ceph-mon[30075]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:11.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1185558213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:19:11.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:11 smithi088 ceph-mon[39202]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:14.292 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:14 smithi088 ceph-mon[30075]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:14.292 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:14 smithi088 ceph-mon[39202]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:14 smithi138 ceph-mon[33706]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:16.340 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:16 smithi088 ceph-mon[39202]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:16.341 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:16 smithi088 ceph-mon[30075]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:16 smithi138 ceph-mon[33706]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T21:19:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T21:19:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T21:19:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:18.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:18 smithi088 ceph-mon[30075]: Deploying daemon osd.3 on smithi088 2024-03-26T21:19:18.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:18 smithi088 ceph-mon[30075]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:18.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:18 smithi088 ceph-mon[39202]: Deploying daemon osd.3 on smithi088 2024-03-26T21:19:18.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:18 smithi088 ceph-mon[39202]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:18 smithi138 ceph-mon[33706]: Deploying daemon osd.3 on smithi088 2024-03-26T21:19:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:18 smithi138 ceph-mon[33706]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:20.446 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:20 smithi088 ceph-mon[39202]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:20.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:20 smithi088 ceph-mon[30075]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:20 smithi138 ceph-mon[33706]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:22 smithi138 ceph-mon[33706]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:22.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:22 smithi088 ceph-mon[30075]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:22.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:22 smithi088 ceph-mon[39202]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:23.309 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:23 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:23.309 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:23 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:23.309 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:23 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:23.309 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:23 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:23 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:23 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:24.253 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:24 smithi088 ceph-mon[30075]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:24.253 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:24 smithi088 ceph-mon[39202]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:24 smithi138 ceph-mon[33706]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:25.922 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 3 on host 'smithi088' 2024-03-26T21:19:25.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:25.928+0000 7efd08960700 1 -- 172.21.15.88:0/3831161993 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 918194574) 0x7efce4000bc0 con 0x7efcf006f4f0 2024-03-26T21:19:25.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:25.930+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 >> v1:172.21.15.88:6800/4097082509 conn(0x7efcf006f4f0 legacy=0x7efcf00719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:25.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:25.930+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 >> v1:172.21.15.88:6790/0 conn(0x7efd04102450 legacy=0x7efd04197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:25.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:25.930+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 shutdown_connections 2024-03-26T21:19:25.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:25.930+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 >> 172.21.15.88:0/3831161993 conn(0x7efd040fd9f0 msgr2=0x7efd0410d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:19:25.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:25.930+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 shutdown_connections 2024-03-26T21:19:25.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:19:25.930+0000 7efd09962700 1 -- 172.21.15.88:0/3831161993 wait complete. 2024-03-26T21:19:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:26.557 DEBUG:teuthology.orchestra.run.smithi088:osd.3> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T21:19:26.560 INFO:tasks.cephadm:Deploying osd.4 on smithi138 with /dev/vg_nvme/lv_4... 2024-03-26T21:19:26.560 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T21:19:26.806 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:19:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:27 smithi138 ceph-mon[33706]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T21:19:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:27 smithi138 ceph-mon[33706]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T21:19:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:27 smithi138 ceph-mon[33706]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[30075]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T21:19:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[30075]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T21:19:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[30075]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[39202]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T21:19:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[39202]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T21:19:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[39202]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-26T21:19:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:27.940 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3[62192]: 2024-03-26T21:19:27.655+0000 7f65a0114700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T21:19:27.940 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:19:27 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3[62192]: 2024-03-26T21:19:27.658+0000 7f6597eff700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:19:28.221 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[30075]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:28.221 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[30075]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:19:28.221 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[30075]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T21:19:28.221 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:28.221 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:28.221 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[30075]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' 2024-03-26T21:19:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:28 smithi138 ceph-mon[33706]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:28 smithi138 ceph-mon[33706]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:19:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:28 smithi138 ceph-mon[33706]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T21:19:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:28 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:28 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:28 smithi138 ceph-mon[33706]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' 2024-03-26T21:19:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[39202]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[39202]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-26T21:19:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[39202]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T21:19:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:28 smithi088 ceph-mon[39202]: from='osd.3 v1:172.21.15.88:6813/3938069994' entity='osd.3' 2024-03-26T21:19:28.685 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T21:19:28.686 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T21:19:28.686 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-26T21:19:28.686 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-26T21:19:28.686 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110264 s, 951 MB/s 2024-03-26T21:19:28.686 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-26T21:19:29.320 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:29 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:19:29.320 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:29 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:19:29.321 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:29 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:29.321 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:29 smithi138 ceph-mon[33706]: osd.3 v1:172.21.15.88:6813/3938069994 boot 2024-03-26T21:19:29.321 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:29 smithi138 ceph-mon[33706]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T21:19:29.321 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:29 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:29.346 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi138:vg_nvme/lv_4 2024-03-26T21:19:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:19:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:19:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[30075]: osd.3 v1:172.21.15.88:6813/3938069994 boot 2024-03-26T21:19:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[30075]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T21:19:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:19:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:19:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[39202]: osd.3 v1:172.21.15.88:6813/3938069994 boot 2024-03-26T21:19:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[39202]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T21:19:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:29 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:19:29.579 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:19:30.450 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:30.450 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.450 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.450 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.451 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.451 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.451 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.451 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.451 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:30.451 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:30.451 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:30 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:30.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:30.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:30 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:31.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- 172.21.15.138:0/4215549405 >> v1:172.21.15.88:6789/0 conn(0x7fc5cc1094a0 legacy=0x7fc5cc10b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:31.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- 172.21.15.138:0/4215549405 shutdown_connections 2024-03-26T21:19:31.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- 172.21.15.138:0/4215549405 >> 172.21.15.138:0/4215549405 conn(0x7fc5cc0fc8d0 msgr2=0x7fc5cc0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:19:31.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- 172.21.15.138:0/4215549405 shutdown_connections 2024-03-26T21:19:31.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- 172.21.15.138:0/4215549405 wait complete. 2024-03-26T21:19:31.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 Processor -- start 2024-03-26T21:19:31.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- start start 2024-03-26T21:19:31.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5cc06b510 con 0x7fc5cc105760 2024-03-26T21:19:31.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5cc06b740 con 0x7fc5cc101330 2024-03-26T21:19:31.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d3793700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5cc1a71b0 con 0x7fc5cc1094a0 2024-03-26T21:19:31.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.247+0000 7fc5d152f700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc5cc101330 0x7fc5cc19cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:52418/0 (socket says 172.21.15.138:52418) 2024-03-26T21:19:31.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.248+0000 7fc5d152f700 1 -- 172.21.15.138:0/750036663 learned_addr learned my addr 172.21.15.138:0/750036663 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:19:31.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348511108 0 0) 0x7fc5cc06b740 con 0x7fc5cc101330 2024-03-26T21:19:31.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5b80035b0 con 0x7fc5cc101330 2024-03-26T21:19:31.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636640119 0 0) 0x7fc5cc06b510 con 0x7fc5cc105760 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5cc06b740 con 0x7fc5cc105760 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 717429958 0 0) 0x7fc5cc1a71b0 con 0x7fc5cc1094a0 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5cc06b510 con 0x7fc5cc1094a0 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1134412571 0 0) 0x7fc5b80035b0 con 0x7fc5cc101330 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5cc1a71b0 con 0x7fc5cc101330 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc5c80017b0 con 0x7fc5cc101330 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4217115836 0 0) 0x7fc5cc06b740 con 0x7fc5cc105760 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5b80035b0 con 0x7fc5cc105760 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc5bc0017e0 con 0x7fc5cc105760 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4005535670 0 0) 0x7fc5cc06b510 con 0x7fc5cc1094a0 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5cc06b740 con 0x7fc5cc1094a0 2024-03-26T21:19:31.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc5c4001950 con 0x7fc5cc1094a0 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 262052634 0 0) 0x7fc5cc1a71b0 con 0x7fc5cc101330 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 >> v1:172.21.15.88:6790/0 conn(0x7fc5cc1094a0 legacy=0x7fc5cc1a3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 >> v1:172.21.15.88:6789/0 conn(0x7fc5cc105760 legacy=0x7fc5cc1a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc1a83d0 con 0x7fc5cc101330 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5cc1a73e0 con 0x7fc5cc101330 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5cc1a7920 con 0x7fc5cc101330 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fc5c8003b30 con 0x7fc5cc101330 2024-03-26T21:19:31.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.250+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc5c8005d00 con 0x7fc5cc101330 2024-03-26T21:19:31.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.251+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b0005150 con 0x7fc5cc101330 2024-03-26T21:19:31.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.252+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7fc5c801c620 con 0x7fc5cc101330 2024-03-26T21:19:31.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.252+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(26..26 src has 1..26) ==== 2803+0+0 (unknown 840828719 0 0) 0x7fc5c8088f00 con 0x7fc5cc101330 2024-03-26T21:19:31.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.257+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fc5c8057050 con 0x7fc5cc101330 2024-03-26T21:19:31.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:31.459+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc5b0000bc0 con 0x7fc5b806f5a0 2024-03-26T21:19:31.454 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:31 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi088 2024-03-26T21:19:31.454 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:31 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi088 to 4016M 2024-03-26T21:19:31.454 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:31 smithi138 ceph-mon[33706]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T21:19:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:31 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi088 2024-03-26T21:19:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:31 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi088 to 4016M 2024-03-26T21:19:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:31 smithi088 ceph-mon[30075]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T21:19:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:31 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi088 2024-03-26T21:19:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:31 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi088 to 4016M 2024-03-26T21:19:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:31 smithi088 ceph-mon[39202]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T21:19:32.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:32 smithi138 ceph-mon[33706]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:32.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:32 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:32.339 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:32 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:32.339 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:32 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[30075]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[39202]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:32.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:32 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:33.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:33 smithi138 ceph-mon[33706]: from='client.24229 v1:172.21.15.138:0/750036663' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:33.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:33 smithi088 ceph-mon[30075]: from='client.24229 v1:172.21.15.138:0/750036663' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:33 smithi088 ceph-mon[39202]: from='client.24229 v1:172.21.15.138:0/750036663' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:34 smithi138 ceph-mon[33706]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/1412432469' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]: dispatch 2024-03-26T21:19:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:34 smithi138 ceph-mon[33706]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]: dispatch 2024-03-26T21:19:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:34 smithi138 ceph-mon[33706]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]': finished 2024-03-26T21:19:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:34 smithi138 ceph-mon[33706]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T21:19:34.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:34 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[30075]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/1412432469' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]: dispatch 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[30075]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]: dispatch 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[30075]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]': finished 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[30075]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[39202]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/1412432469' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]: dispatch 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[39202]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]: dispatch 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[39202]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2a562d8-4071-44c7-9695-7652271cbd52"}]': finished 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[39202]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T21:19:34.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:34 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/2731784248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:19:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/2731784248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:19:35.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/2731784248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:19:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:36 smithi138 ceph-mon[33706]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:36 smithi088 ceph-mon[30075]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:36.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:36 smithi088 ceph-mon[39202]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:37.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:37 smithi138 ceph-mon[33706]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:37.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:37 smithi088 ceph-mon[30075]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:37 smithi088 ceph-mon[39202]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:40.437 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:40 smithi088 ceph-mon[30075]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:40.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:40 smithi088 ceph-mon[39202]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:40.448 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:40 smithi138 ceph-mon[33706]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:41.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:41 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T21:19:41.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:41 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:41.454 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:41 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T21:19:41.454 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:41 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:41 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T21:19:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:41 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:42.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:42 smithi138 ceph-mon[33706]: Deploying daemon osd.4 on smithi138 2024-03-26T21:19:42.339 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:42 smithi138 ceph-mon[33706]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:42.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:42 smithi088 ceph-mon[30075]: Deploying daemon osd.4 on smithi138 2024-03-26T21:19:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:42 smithi088 ceph-mon[30075]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:42 smithi088 ceph-mon[39202]: Deploying daemon osd.4 on smithi138 2024-03-26T21:19:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:42 smithi088 ceph-mon[39202]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:44.381 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:44 smithi138 ceph-mon[33706]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:44 smithi088 ceph-mon[30075]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:44 smithi088 ceph-mon[39202]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:46 smithi138 ceph-mon[33706]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:46 smithi088 ceph-mon[30075]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:46 smithi088 ceph-mon[39202]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:47.569 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:47 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:47.569 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:47 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:47.569 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:47 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:47.569 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:47 smithi138 ceph-mon[33706]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[30075]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:47 smithi088 ceph-mon[39202]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:50.340 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:50.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: from='osd.4 v1:172.21.15.138:6800/2187218446' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:19:50.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:19:50.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:50.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:50.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:50 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.421 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 4 on host 'smithi138' 2024-03-26T21:19:50.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:50.427+0000 7fc5c27fc700 1 -- 172.21.15.138:0/750036663 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 189330176) 0x7fc5b0000bc0 con 0x7fc5b806f5a0 2024-03-26T21:19:50.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:50.429+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 >> v1:172.21.15.88:6800/4097082509 conn(0x7fc5b806f5a0 legacy=0x7fc5b8071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:50.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:50.429+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 >> v1:172.21.15.138:6789/0 conn(0x7fc5cc101330 legacy=0x7fc5cc19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:50.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:50.429+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 shutdown_connections 2024-03-26T21:19:50.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:50.429+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 >> 172.21.15.138:0/750036663 conn(0x7fc5cc0fc8d0 msgr2=0x7fc5cc10c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:19:50.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:50.429+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 shutdown_connections 2024-03-26T21:19:50.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:50.429+0000 7fc5d3793700 1 -- 172.21.15.138:0/750036663 wait complete. 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: from='osd.4 v1:172.21.15.138:6800/2187218446' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: from='osd.4 v1:172.21.15.138:6800/2187218446' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:50 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:51.111 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:19:51.111 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: from='osd.4 v1:172.21.15.138:6800/2187218446' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:19:51.111 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T21:19:51.112 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:51.112 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:19:51.112 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:51.112 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:51.112 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:51.137 DEBUG:teuthology.orchestra.run.smithi138:osd.4> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T21:19:51.139 INFO:tasks.cephadm:Deploying osd.5 on smithi138 with /dev/vg_nvme/lv_3... 2024-03-26T21:19:51.140 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T21:19:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:19:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: from='osd.4 v1:172.21.15.138:6800/2187218446' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:19:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T21:19:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:19:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: from='osd.4 v1:172.21.15.138:6800/2187218446' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:19:51.896 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:19:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:19:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T21:19:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: osd.4 v1:172.21.15.138:6800/2187218446 boot 2024-03-26T21:19:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T21:19:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:52 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: osd.4 v1:172.21.15.138:6800/2187218446 boot 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:19:52.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: osd.4 v1:172.21.15.138:6800/2187218446 boot 2024-03-26T21:19:52.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T21:19:52.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:52 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:19:53.085 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:53 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:19:53.085 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:53 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:19:53.085 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:53 smithi138 ceph-mon[33706]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T21:19:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:53 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:19:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:53 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:19:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:53 smithi088 ceph-mon[30075]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T21:19:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:53 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:19:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:53 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:19:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:53 smithi088 ceph-mon[39202]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T21:19:54.201 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:54 smithi138 ceph-mon[33706]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:54.201 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:54 smithi138 ceph-mon[33706]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T21:19:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:54 smithi088 ceph-mon[30075]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:54 smithi088 ceph-mon[30075]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T21:19:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:54 smithi088 ceph-mon[39202]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:54 smithi088 ceph-mon[39202]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T21:19:54.971 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T21:19:54.972 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T21:19:54.972 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-26T21:19:54.972 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-26T21:19:54.972 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110017 s, 953 MB/s 2024-03-26T21:19:54.972 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-26T21:19:55.624 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi138:vg_nvme/lv_3 2024-03-26T21:19:55.861 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:19:55.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T21:19:55.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi138 2024-03-26T21:19:55.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:55.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi138 to 17088M 2024-03-26T21:19:55.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:55.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:55.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:55.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:55 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi138 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi138 to 17088M 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi138 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi138 to 17088M 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:55 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:19:56.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:56 smithi138 ceph-mon[33706]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T21:19:56.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:56 smithi088 ceph-mon[30075]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T21:19:56.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:56 smithi088 ceph-mon[39202]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T21:19:57.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.551+0000 7fb5cd454700 1 -- 172.21.15.138:0/4273077190 >> v1:172.21.15.138:6789/0 conn(0x7fb5c8106a00 legacy=0x7fb5c8108df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:57.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.551+0000 7fb5cd454700 1 -- 172.21.15.138:0/4273077190 shutdown_connections 2024-03-26T21:19:57.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.551+0000 7fb5cd454700 1 -- 172.21.15.138:0/4273077190 >> 172.21.15.138:0/4273077190 conn(0x7fb5c80fc900 msgr2=0x7fb5c80fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:19:57.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.551+0000 7fb5cd454700 1 -- 172.21.15.138:0/4273077190 shutdown_connections 2024-03-26T21:19:57.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.551+0000 7fb5cd454700 1 -- 172.21.15.138:0/4273077190 wait complete. 2024-03-26T21:19:57.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5cd454700 1 Processor -- start 2024-03-26T21:19:57.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5cd454700 1 -- start start 2024-03-26T21:19:57.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5cd454700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5c81a4cf0 con 0x7fb5c8102ef0 2024-03-26T21:19:57.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5cd454700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5c81a5f30 con 0x7fb5c8106a00 2024-03-26T21:19:57.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5cd454700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb5c81a7170 con 0x7fb5c810a5a0 2024-03-26T21:19:57.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5be7fc700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fb5c8106a00 0x7fb5c819fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:40660/0 (socket says 172.21.15.138:40660) 2024-03-26T21:19:57.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5be7fc700 1 -- 172.21.15.138:0/3918109826 learned_addr learned my addr 172.21.15.138:0/3918109826 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:19:57.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162058043 0 0) 0x7fb5c81a5f30 con 0x7fb5c8106a00 2024-03-26T21:19:57.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5b40035b0 con 0x7fb5c8106a00 2024-03-26T21:19:57.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298553923 0 0) 0x7fb5c81a4cf0 con 0x7fb5c8102ef0 2024-03-26T21:19:57.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5c81a5f30 con 0x7fb5c8102ef0 2024-03-26T21:19:57.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3412846446 0 0) 0x7fb5c81a7170 con 0x7fb5c810a5a0 2024-03-26T21:19:57.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.552+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5c81a4cf0 con 0x7fb5c810a5a0 2024-03-26T21:19:57.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1568731594 0 0) 0x7fb5b40035b0 con 0x7fb5c8106a00 2024-03-26T21:19:57.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5c81a7170 con 0x7fb5c8106a00 2024-03-26T21:19:57.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb5b0001430 con 0x7fb5c8106a00 2024-03-26T21:19:57.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1645870162 0 0) 0x7fb5c81a5f30 con 0x7fb5c8102ef0 2024-03-26T21:19:57.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5b40035b0 con 0x7fb5c8102ef0 2024-03-26T21:19:57.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb5b80018b0 con 0x7fb5c8102ef0 2024-03-26T21:19:57.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3731125586 0 0) 0x7fb5c81a7170 con 0x7fb5c8106a00 2024-03-26T21:19:57.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 >> v1:172.21.15.88:6790/0 conn(0x7fb5c810a5a0 legacy=0x7fb5c81a33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:57.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 >> v1:172.21.15.88:6789/0 conn(0x7fb5c8102ef0 legacy=0x7fb5c8102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:19:57.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c81a83b0 con 0x7fb5c8106a00 2024-03-26T21:19:57.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c81a4f20 con 0x7fb5c8106a00 2024-03-26T21:19:57.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5c81a5460 con 0x7fb5c8106a00 2024-03-26T21:19:57.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fb5b0003720 con 0x7fb5c8106a00 2024-03-26T21:19:57.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.553+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb5b0004fe0 con 0x7fb5c8106a00 2024-03-26T21:19:57.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.554+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a8005150 con 0x7fb5c8106a00 2024-03-26T21:19:57.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.554+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7fb5b0003ab0 con 0x7fb5c8106a00 2024-03-26T21:19:57.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.555+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(34..34 src has 1..34) ==== 3205+0+0 (unknown 660940352 0 0) 0x7fb5b0089200 con 0x7fb5c8106a00 2024-03-26T21:19:57.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.559+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fb5b0004fc0 con 0x7fb5c8106a00 2024-03-26T21:19:57.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:19:57.769+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb5a8000bc0 con 0x7fb5b406f1f0 2024-03-26T21:19:57.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:57 smithi138 ceph-mon[33706]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:57.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:57 smithi088 ceph-mon[30075]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:57.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:57 smithi088 ceph-mon[39202]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:19:58.949 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:58 smithi138 ceph-mon[33706]: from='client.24256 v1:172.21.15.138:0/3918109826' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:58.949 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:58 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:58.949 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:58 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:58.949 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:58 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[30075]: from='client.24256 v1:172.21.15.138:0/3918109826' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[39202]: from='client.24256 v1:172.21.15.138:0/3918109826' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:19:58.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:58 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:19:59.959 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:19:59 smithi138 ceph-mon[33706]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:19:59.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:19:59 smithi088 ceph-mon[30075]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:19:59.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:19:59 smithi088 ceph-mon[39202]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:20:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:00 smithi138 ceph-mon[33706]: overall HEALTH_OK 2024-03-26T21:20:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/755795208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]: dispatch 2024-03-26T21:20:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:00 smithi138 ceph-mon[33706]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]: dispatch 2024-03-26T21:20:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:00 smithi138 ceph-mon[33706]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]': finished 2024-03-26T21:20:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:00 smithi138 ceph-mon[33706]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T21:20:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:00 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:00.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[30075]: overall HEALTH_OK 2024-03-26T21:20:00.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/755795208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]: dispatch 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[30075]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]: dispatch 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[30075]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]': finished 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[30075]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[39202]: overall HEALTH_OK 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/755795208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]: dispatch 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[39202]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]: dispatch 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[39202]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c2c5207-7f0a-4700-b8e1-afeba8499378"}]': finished 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[39202]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T21:20:00.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:00 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:01.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:01 smithi138 ceph-mon[33706]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-26T21:20:01.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/37880894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:01.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:01 smithi088 ceph-mon[39202]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-26T21:20:01.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/37880894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:01.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:01 smithi088 ceph-mon[30075]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-26T21:20:01.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/37880894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:04.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:04 smithi138 ceph-mon[33706]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:20:04.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:04 smithi088 ceph-mon[39202]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:20:04.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:04 smithi088 ceph-mon[30075]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:20:06.336 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:06 smithi138 ceph-mon[33706]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-26T21:20:06.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:06 smithi088 ceph-mon[30075]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-26T21:20:06.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:06 smithi088 ceph-mon[39202]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-26T21:20:08.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:08 smithi138 ceph-mon[33706]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:20:08.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:08 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T21:20:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:08 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:08.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:08 smithi138 ceph-mon[33706]: Deploying daemon osd.5 on smithi138 2024-03-26T21:20:08.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[30075]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:20:08.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T21:20:08.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:08.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[30075]: Deploying daemon osd.5 on smithi138 2024-03-26T21:20:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[39202]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:20:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T21:20:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:08 smithi088 ceph-mon[39202]: Deploying daemon osd.5 on smithi138 2024-03-26T21:20:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:10 smithi138 ceph-mon[33706]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:10.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:10 smithi088 ceph-mon[30075]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:10.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:10 smithi088 ceph-mon[39202]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:12.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:12 smithi138 ceph-mon[33706]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:12.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:12 smithi088 ceph-mon[30075]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:12.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:12 smithi088 ceph-mon[39202]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:13 smithi138 ceph-mon[33706]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[30075]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:13 smithi088 ceph-mon[39202]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:16.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:16 smithi138 ceph-mon[33706]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:16.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:16 smithi138 ceph-mon[33706]: from='osd.5 v1:172.21.15.138:6804/2329820906' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:20:16.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:16 smithi138 ceph-mon[33706]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:20:16.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:16.481+0000 7fb5c4ff9700 1 -- 172.21.15.138:0/3918109826 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 1465720244) 0x7fb5a8000bc0 con 0x7fb5b406f1f0 2024-03-26T21:20:16.475 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 5 on host 'smithi138' 2024-03-26T21:20:16.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:16.483+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 >> v1:172.21.15.88:6800/4097082509 conn(0x7fb5b406f1f0 legacy=0x7fb5b40716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:16.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:16.483+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 >> v1:172.21.15.138:6789/0 conn(0x7fb5c8106a00 legacy=0x7fb5c819fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:16.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:16.483+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 shutdown_connections 2024-03-26T21:20:16.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:16.483+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 >> 172.21.15.138:0/3918109826 conn(0x7fb5c80fc900 msgr2=0x7fb5c810ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:20:16.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:16.483+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 shutdown_connections 2024-03-26T21:20:16.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:16.483+0000 7fb5cd454700 1 -- 172.21.15.138:0/3918109826 wait complete. 2024-03-26T21:20:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:16 smithi088 ceph-mon[30075]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:16 smithi088 ceph-mon[30075]: from='osd.5 v1:172.21.15.138:6804/2329820906' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:20:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:16 smithi088 ceph-mon[30075]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:20:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:16 smithi088 ceph-mon[39202]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:16 smithi088 ceph-mon[39202]: from='osd.5 v1:172.21.15.138:6804/2329820906' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:20:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:16 smithi088 ceph-mon[39202]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:20:17.239 DEBUG:teuthology.orchestra.run.smithi138:osd.5> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T21:20:17.241 INFO:tasks.cephadm:Deploying osd.6 on smithi138 with /dev/vg_nvme/lv_2... 2024-03-26T21:20:17.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T21:20:17.446 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T21:20:17.446 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='osd.5 v1:172.21.15.138:6804/2329820906' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T21:20:17.447 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='osd.5 v1:172.21.15.138:6804/2329820906' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T21:20:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='osd.5 v1:172.21.15.138:6804/2329820906' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T21:20:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:17 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:17.712 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:20:17.590+0000 7fe0b5340700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T21:20:17.712 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:20:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:20:17.595+0000 7fe0ad12b700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:20:18.063 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:20:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:18 smithi138 ceph-mon[33706]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:18 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:18 smithi138 ceph-mon[33706]: from='osd.5 ' entity='osd.5' 2024-03-26T21:20:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:18 smithi138 ceph-mon[33706]: osd.5 v1:172.21.15.138:6804/2329820906 boot 2024-03-26T21:20:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:18 smithi138 ceph-mon[33706]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T21:20:18.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:18 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[30075]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[30075]: from='osd.5 ' entity='osd.5' 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[30075]: osd.5 v1:172.21.15.138:6804/2329820906 boot 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[30075]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[39202]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[39202]: from='osd.5 ' entity='osd.5' 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[39202]: osd.5 v1:172.21.15.138:6804/2329820906 boot 2024-03-26T21:20:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[39202]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T21:20:18.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:18 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:20:19.212 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:19 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:20:19.212 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:19 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:20:19.212 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:19 smithi138 ceph-mon[33706]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T21:20:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:19 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:20:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:19 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:20:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:19 smithi088 ceph-mon[39202]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T21:20:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:19 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:20:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:19 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:20:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:19 smithi088 ceph-mon[30075]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T21:20:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:20 smithi138 ceph-mon[33706]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:20 smithi138 ceph-mon[33706]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T21:20:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:20 smithi088 ceph-mon[30075]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:20 smithi088 ceph-mon[30075]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T21:20:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:20 smithi088 ceph-mon[39202]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:20.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:20 smithi088 ceph-mon[39202]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T21:20:21.155 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T21:20:21.155 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T21:20:21.155 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-26T21:20:21.155 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-26T21:20:21.155 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109682 s, 956 MB/s 2024-03-26T21:20:21.155 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-26T21:20:21.940 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi138:vg_nvme/lv_2 2024-03-26T21:20:22.175 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:20:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:22.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:22.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:22.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:22.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:22.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:22.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:22.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:23.353 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:23 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi138 2024-03-26T21:20:23.354 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:23 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi138 to 8544M 2024-03-26T21:20:23.354 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:23 smithi138 ceph-mon[33706]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T21:20:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:23 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi138 2024-03-26T21:20:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:23 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi138 to 8544M 2024-03-26T21:20:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:23 smithi088 ceph-mon[39202]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T21:20:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:23 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi138 2024-03-26T21:20:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:23 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi138 to 8544M 2024-03-26T21:20:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:23 smithi088 ceph-mon[30075]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.862+0000 7fae3ef58700 1 -- 172.21.15.138:0/3782107471 >> v1:172.21.15.138:6789/0 conn(0x7fae381068b0 legacy=0x7fae38108d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.862+0000 7fae3ef58700 1 -- 172.21.15.138:0/3782107471 shutdown_connections 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.862+0000 7fae3ef58700 1 -- 172.21.15.138:0/3782107471 >> 172.21.15.138:0/3782107471 conn(0x7fae380fda40 msgr2=0x7fae380ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.862+0000 7fae3ef58700 1 -- 172.21.15.138:0/3782107471 shutdown_connections 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.862+0000 7fae3ef58700 1 -- 172.21.15.138:0/3782107471 wait complete. 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae3ef58700 1 Processor -- start 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae3ef58700 1 -- start start 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae3ef58700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fae38198190 con 0x7fae3810a5f0 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae3ef58700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fae381a3980 con 0x7fae381068b0 2024-03-26T21:20:23.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae3ef58700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fae381a4b60 con 0x7fae38102480 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae37fff700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fae381068b0 0x7fae3819db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33020/0 (socket says 172.21.15.138:33020) 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae37fff700 1 -- 172.21.15.138:0/2006568618 learned_addr learned my addr 172.21.15.138:0/2006568618 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.863+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3182324421 0 0) 0x7fae381a3980 con 0x7fae381068b0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae280035b0 con 0x7fae381068b0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338400865 0 0) 0x7fae381a4b60 con 0x7fae38102480 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae381a3980 con 0x7fae38102480 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595277833 0 0) 0x7fae38198190 con 0x7fae3810a5f0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae381a4b60 con 0x7fae3810a5f0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1279943684 0 0) 0x7fae280035b0 con 0x7fae381068b0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae38198190 con 0x7fae381068b0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fae24004740 con 0x7fae381068b0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1534591223 0 0) 0x7fae38198190 con 0x7fae381068b0 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 >> v1:172.21.15.88:6790/0 conn(0x7fae38102480 legacy=0x7fae38197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 >> v1:172.21.15.88:6789/0 conn(0x7fae3810a5f0 legacy=0x7fae381a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:23.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae381a5d80 con 0x7fae381068b0 2024-03-26T21:20:23.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fae381a4d90 con 0x7fae381068b0 2024-03-26T21:20:23.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fae24001bf0 con 0x7fae381068b0 2024-03-26T21:20:23.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fae240052f0 con 0x7fae381068b0 2024-03-26T21:20:23.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.864+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fae381a5250 con 0x7fae381068b0 2024-03-26T21:20:23.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.865+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae38068f30 con 0x7fae381068b0 2024-03-26T21:20:23.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.865+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7fae24003940 con 0x7fae381068b0 2024-03-26T21:20:23.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.866+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3497+0+0 (unknown 1681576915 0 0) 0x7fae24089780 con 0x7fae381068b0 2024-03-26T21:20:23.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:23.871+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fae240575c0 con 0x7fae381068b0 2024-03-26T21:20:24.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:24.077+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fae38107f60 con 0x7fae2806f5d0 2024-03-26T21:20:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:24 smithi138 ceph-mon[33706]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:24 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:20:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:24 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:20:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:24 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[30075]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[39202]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:20:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:24 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:25 smithi138 ceph-mon[33706]: from='client.24283 v1:172.21.15.138:0/2006568618' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:20:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:25 smithi088 ceph-mon[30075]: from='client.24283 v1:172.21.15.138:0/2006568618' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:20:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:25 smithi088 ceph-mon[39202]: from='client.24283 v1:172.21.15.138:0/2006568618' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:20:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:26 smithi138 ceph-mon[33706]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:26 smithi088 ceph-mon[30075]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:26 smithi088 ceph-mon[39202]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/1849139000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]: dispatch 2024-03-26T21:20:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:27 smithi138 ceph-mon[33706]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]: dispatch 2024-03-26T21:20:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:27 smithi138 ceph-mon[33706]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]': finished 2024-03-26T21:20:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:27 smithi138 ceph-mon[33706]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T21:20:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/1849139000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]: dispatch 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[39202]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]: dispatch 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[39202]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]': finished 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[39202]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/1849139000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]: dispatch 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[30075]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]: dispatch 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[30075]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dbb6451-7d30-4039-9b3d-f01e9acd00c7"}]': finished 2024-03-26T21:20:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[30075]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T21:20:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:28.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:28 smithi138 ceph-mon[33706]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:28.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/4245570936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:28 smithi088 ceph-mon[39202]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/4245570936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:28 smithi088 ceph-mon[30075]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/4245570936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:30.390 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:30 smithi138 ceph-mon[33706]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:30 smithi088 ceph-mon[30075]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:30 smithi088 ceph-mon[39202]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:32.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:32 smithi138 ceph-mon[33706]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:32.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:32 smithi088 ceph-mon[30075]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:32 smithi088 ceph-mon[39202]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:34.372 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:34 smithi138 ceph-mon[33706]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:34.372 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:34 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T21:20:34.372 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:34 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:34 smithi088 ceph-mon[30075]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:34 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T21:20:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:34 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:34 smithi088 ceph-mon[39202]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:34 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T21:20:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:34 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:35.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:35 smithi138 ceph-mon[33706]: Deploying daemon osd.6 on smithi138 2024-03-26T21:20:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:35 smithi088 ceph-mon[30075]: Deploying daemon osd.6 on smithi138 2024-03-26T21:20:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:35 smithi088 ceph-mon[39202]: Deploying daemon osd.6 on smithi138 2024-03-26T21:20:36.384 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:36 smithi138 ceph-mon[33706]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:36.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:36 smithi088 ceph-mon[30075]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:36 smithi088 ceph-mon[39202]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:38.459 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:38 smithi138 ceph-mon[33706]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:38.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:38 smithi088 ceph-mon[30075]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:38 smithi088 ceph-mon[39202]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:40 smithi138 ceph-mon[33706]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:40.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:40 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[39202]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[30075]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:40 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:42 smithi138 ceph-mon[33706]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:42.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:42.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[30075]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:42.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[39202]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:42.883 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:42.890+0000 7fae35ffb700 1 -- 172.21.15.138:0/2006568618 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3009621608) 0x7fae38107f60 con 0x7fae2806f5d0 2024-03-26T21:20:42.884 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 6 on host 'smithi138' 2024-03-26T21:20:42.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:42.893+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 >> v1:172.21.15.88:6800/4097082509 conn(0x7fae2806f5d0 legacy=0x7fae28071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:42.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:42.893+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 >> v1:172.21.15.138:6789/0 conn(0x7fae381068b0 legacy=0x7fae3819db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:42.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:42.893+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 shutdown_connections 2024-03-26T21:20:42.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:42.893+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 >> 172.21.15.138:0/2006568618 conn(0x7fae380fda40 msgr2=0x7fae38109bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:20:42.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:42.893+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 shutdown_connections 2024-03-26T21:20:42.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:42.893+0000 7fae3ef58700 1 -- 172.21.15.138:0/2006568618 wait complete. 2024-03-26T21:20:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:43 smithi138 ceph-mon[33706]: from='osd.6 v1:172.21.15.138:6808/771295543' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:20:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:43 smithi138 ceph-mon[33706]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:20:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[30075]: from='osd.6 v1:172.21.15.138:6808/771295543' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:20:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[30075]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:20:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[39202]: from='osd.6 v1:172.21.15.138:6808/771295543' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:20:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[39202]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:20:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:20:43.515 DEBUG:teuthology.orchestra.run.smithi138:osd.6> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T21:20:43.517 INFO:tasks.cephadm:Deploying osd.7 on smithi138 with /dev/vg_nvme/lv_1... 2024-03-26T21:20:43.518 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T21:20:44.427 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-mon[33706]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:44.427 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-mon[33706]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T21:20:44.427 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-mon[33706]: from='osd.6 v1:172.21.15.138:6808/771295543' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:44.427 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-mon[33706]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T21:20:44.428 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:44.428 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-mon[33706]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:44.428 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:20:44.180+0000 7f4266274700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T21:20:44.428 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:20:44 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:20:44.186+0000 7f4262067700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:20:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[30075]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[30075]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T21:20:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[30075]: from='osd.6 v1:172.21.15.138:6808/771295543' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[30075]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T21:20:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[30075]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[39202]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[39202]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T21:20:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[39202]: from='osd.6 v1:172.21.15.138:6808/771295543' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[39202]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T21:20:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:44 smithi088 ceph-mon[39202]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:20:44.732 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:20:45.251 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:45 smithi138 ceph-mon[33706]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:20:45.251 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:45 smithi138 ceph-mon[33706]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T21:20:45.251 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:45 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:45.252 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:45 smithi138 ceph-mon[33706]: osd.6 v1:172.21.15.138:6808/771295543 boot 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[30075]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[30075]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[30075]: osd.6 v1:172.21.15.138:6808/771295543 boot 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[39202]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[39202]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:45 smithi088 ceph-mon[39202]: osd.6 v1:172.21.15.138:6808/771295543 boot 2024-03-26T21:20:46.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:46 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:20:46.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:46 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:20:46.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:46 smithi138 ceph-mon[33706]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:46.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:46 smithi138 ceph-mon[33706]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T21:20:46.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:46 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:46.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:46 smithi138 ceph-mon[33706]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T21:20:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[39202]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[39202]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[39202]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[30075]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:20:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[30075]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T21:20:46.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:20:46.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:46 smithi088 ceph-mon[30075]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T21:20:47.867 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T21:20:47.867 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T21:20:47.867 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-26T21:20:47.867 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-26T21:20:47.867 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111656 s, 939 MB/s 2024-03-26T21:20:47.867 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-26T21:20:48.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi138 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi138 to 5696M 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:20:48.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:48 smithi138 ceph-mon[33706]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi138 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi138 to 5696M 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:20:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[30075]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi138 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi138 to 5696M 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:20:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:20:48.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:48 smithi088 ceph-mon[39202]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T21:20:48.487 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi138:vg_nvme/lv_1 2024-03-26T21:20:48.725 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:20:49.370 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:49 smithi138 ceph-mon[33706]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T21:20:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:49 smithi088 ceph-mon[39202]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T21:20:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:49 smithi088 ceph-mon[30075]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T21:20:50.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:50 smithi138 ceph-mon[33706]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:20:50.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:50 smithi138 ceph-mon[33706]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T21:20:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:50 smithi088 ceph-mon[30075]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:20:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:50 smithi088 ceph-mon[30075]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T21:20:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:50 smithi088 ceph-mon[39202]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:20:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:50 smithi088 ceph-mon[39202]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T21:20:50.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.556+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/1730809955 >> v1:172.21.15.138:6789/0 conn(0x7fd4cc105770 legacy=0x7fd4cc107bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:50.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.556+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/1730809955 shutdown_connections 2024-03-26T21:20:50.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.556+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/1730809955 >> 172.21.15.138:0/1730809955 conn(0x7fd4cc0fc920 msgr2=0x7fd4cc0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:20:50.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.556+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/1730809955 shutdown_connections 2024-03-26T21:20:50.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/1730809955 wait complete. 2024-03-26T21:20:50.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4d0fbf700 1 Processor -- start 2024-03-26T21:20:50.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4d0fbf700 1 -- start start 2024-03-26T21:20:50.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4d0fbf700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4cc06b870 con 0x7fd4cc101340 2024-03-26T21:20:50.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4d0fbf700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4cc1a3ca0 con 0x7fd4cc1094b0 2024-03-26T21:20:50.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4d0fbf700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4cc1a4ec0 con 0x7fd4cc105770 2024-03-26T21:20:50.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4cad9d700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fd4cc1094b0 0x7fd4cc1a1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47598/0 (socket says 172.21.15.138:47598) 2024-03-26T21:20:50.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.557+0000 7fd4cad9d700 1 -- 172.21.15.138:0/3042788277 learned_addr learned my addr 172.21.15.138:0/3042788277 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:20:50.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764402255 0 0) 0x7fd4cc1a3ca0 con 0x7fd4cc1094b0 2024-03-26T21:20:50.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4b40035b0 con 0x7fd4cc1094b0 2024-03-26T21:20:50.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 756437765 0 0) 0x7fd4cc1a4ec0 con 0x7fd4cc105770 2024-03-26T21:20:50.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4cc1a3ca0 con 0x7fd4cc105770 2024-03-26T21:20:50.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 669378337 0 0) 0x7fd4cc06b870 con 0x7fd4cc101340 2024-03-26T21:20:50.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4cc1a4ec0 con 0x7fd4cc101340 2024-03-26T21:20:50.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1550501193 0 0) 0x7fd4b40035b0 con 0x7fd4cc1094b0 2024-03-26T21:20:50.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4cc06b870 con 0x7fd4cc1094b0 2024-03-26T21:20:50.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd4c4001c70 con 0x7fd4cc1094b0 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2816533051 0 0) 0x7fd4cc1a3ca0 con 0x7fd4cc105770 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4b40035b0 con 0x7fd4cc105770 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd4c0004550 con 0x7fd4cc105770 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3096411748 0 0) 0x7fd4cc1a4ec0 con 0x7fd4cc101340 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4cc1a3ca0 con 0x7fd4cc101340 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd4bc0017b0 con 0x7fd4cc101340 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2227701736 0 0) 0x7fd4cc06b870 con 0x7fd4cc1094b0 2024-03-26T21:20:50.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 >> v1:172.21.15.88:6790/0 conn(0x7fd4cc105770 legacy=0x7fd4cc19de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 >> v1:172.21.15.88:6789/0 conn(0x7fd4cc101340 legacy=0x7fd4cc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4cc1a6100 con 0x7fd4cc1094b0 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4cc1a2cf0 con 0x7fd4cc1094b0 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd4cc1a3230 con 0x7fd4cc1094b0 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fd4c4003eb0 con 0x7fd4cc1094b0 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.558+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd4c4005fc0 con 0x7fd4cc1094b0 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.559+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ac005150 con 0x7fd4cc1094b0 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.560+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7fd4c4003a20 con 0x7fd4cc1094b0 2024-03-26T21:20:50.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.560+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(50..50 src has 1..50) ==== 3789+0+0 (unknown 3044325398 0 0) 0x7fd4c4089770 con 0x7fd4cc1094b0 2024-03-26T21:20:50.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.565+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fd4c4057490 con 0x7fd4cc1094b0 2024-03-26T21:20:50.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:20:50.771+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd4ac000bc0 con 0x7fd4b406f570 2024-03-26T21:20:51.461 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:20:51.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:20:51.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:51 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:20:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:20:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:51 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:20:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:20:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:51 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:20:52.334 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:52 smithi138 ceph-mon[33706]: from='client.24310 v1:172.21.15.138:0/3042788277' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:20:52.334 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:52 smithi138 ceph-mon[33706]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T21:20:52.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:52 smithi088 ceph-mon[30075]: from='client.24310 v1:172.21.15.138:0/3042788277' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:20:52.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:52 smithi088 ceph-mon[30075]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T21:20:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:52 smithi088 ceph-mon[39202]: from='client.24310 v1:172.21.15.138:0/3042788277' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:20:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:52 smithi088 ceph-mon[39202]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T21:20:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:54 smithi138 ceph-mon[33706]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:20:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/932657705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]: dispatch 2024-03-26T21:20:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:54 smithi138 ceph-mon[33706]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]: dispatch 2024-03-26T21:20:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:54 smithi138 ceph-mon[33706]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]': finished 2024-03-26T21:20:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:54 smithi138 ceph-mon[33706]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T21:20:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:54 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:20:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/3421691552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[30075]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/932657705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]: dispatch 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[30075]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]: dispatch 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[30075]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]': finished 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[30075]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/3421691552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[39202]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/932657705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]: dispatch 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[39202]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]: dispatch 2024-03-26T21:20:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[39202]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0ba8a7d-0c56-4b23-838a-866cad1b91a6"}]': finished 2024-03-26T21:20:54.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[39202]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T21:20:54.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:20:54.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/3421691552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:20:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:56 smithi138 ceph-mon[33706]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T21:20:56.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:56 smithi088 ceph-mon[39202]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T21:20:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:56 smithi088 ceph-mon[30075]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T21:20:58.466 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:20:58 smithi138 ceph-mon[33706]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:20:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:20:58 smithi088 ceph-mon[39202]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:20:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:20:58 smithi088 ceph-mon[30075]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:21:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:00 smithi138 ceph-mon[33706]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:21:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:00 smithi088 ceph-mon[39202]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:21:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:00 smithi088 ceph-mon[30075]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:21:01.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:01 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T21:21:01.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:01 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:01.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:01 smithi138 ceph-mon[33706]: Deploying daemon osd.7 on smithi138 2024-03-26T21:21:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:01 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T21:21:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:01 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:01 smithi088 ceph-mon[39202]: Deploying daemon osd.7 on smithi138 2024-03-26T21:21:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:01 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T21:21:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:01 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:01 smithi088 ceph-mon[30075]: Deploying daemon osd.7 on smithi138 2024-03-26T21:21:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:02 smithi138 ceph-mon[33706]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:02.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:02 smithi088 ceph-mon[39202]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:02.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:02 smithi088 ceph-mon[30075]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:04.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:04 smithi138 ceph-mon[33706]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:04.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:04 smithi088 ceph-mon[30075]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:04.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:04 smithi088 ceph-mon[39202]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:06.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:06 smithi138 ceph-mon[33706]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:06.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:06.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:06.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:06 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:06.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[30075]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:06.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:06.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:06.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:06.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[39202]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:06.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:06.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:06.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:06 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:08.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:08 smithi138 ceph-mon[33706]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:08 smithi088 ceph-mon[39202]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:08.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:08 smithi088 ceph-mon[30075]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:09 smithi138 ceph-mon[33706]: from='osd.7 v1:172.21.15.138:6812/1004436505' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:21:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:09 smithi138 ceph-mon[33706]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:21:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:09 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:09 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[39202]: from='osd.7 v1:172.21.15.138:6812/1004436505' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:21:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[39202]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:21:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:09.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:09.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[30075]: from='osd.7 v1:172.21.15.138:6812/1004436505' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:21:09.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[30075]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:21:09.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:09.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:09 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:09.880 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 7 on host 'smithi138' 2024-03-26T21:21:09.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:09.886+0000 7fd4bb7fe700 1 -- 172.21.15.138:0/3042788277 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 4017537244) 0x7fd4ac000bc0 con 0x7fd4b406f570 2024-03-26T21:21:09.883 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:09.889+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 >> v1:172.21.15.88:6800/4097082509 conn(0x7fd4b406f570 legacy=0x7fd4b4071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:09.883 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:09.889+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 >> v1:172.21.15.138:6789/0 conn(0x7fd4cc1094b0 legacy=0x7fd4cc1a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:09.883 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:09.889+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 shutdown_connections 2024-03-26T21:21:09.883 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:09.889+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 >> 172.21.15.138:0/3042788277 conn(0x7fd4cc0fc920 msgr2=0x7fd4cc107bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:09.883 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:09.889+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 shutdown_connections 2024-03-26T21:21:09.883 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:09.889+0000 7fd4d0fbf700 1 -- 172.21.15.138:0/3042788277 wait complete. 2024-03-26T21:21:10.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:10.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:21:10.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: from='osd.7 v1:172.21.15.138:6812/1004436505' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:21:10.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T21:21:10.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:10.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:21:10.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:10.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:10 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:10.545 DEBUG:teuthology.orchestra.run.smithi138:osd.7> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T21:21:10.547 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T21:21:10.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T21:21:10.577 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:10.577 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:21:10.577 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: from='osd.7 v1:172.21.15.138:6812/1004436505' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:21:10.577 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T21:21:10.577 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: from='osd.7 v1:172.21.15.138:6812/1004436505' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:10.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:10 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:10.797 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:21:11.454 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:11 smithi138 ceph-mon[33706]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:21:11.455 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:11 smithi138 ceph-mon[33706]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T21:21:11.455 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:11 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:11.455 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:11 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[30075]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:21:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[30075]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T21:21:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[39202]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-26T21:21:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[39202]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T21:21:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:11 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:12.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: purged_snaps scrub starts 2024-03-26T21:21:12.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: purged_snaps scrub ok 2024-03-26T21:21:12.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:12.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:12.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: osd.7 v1:172.21.15.138:6812/1004436505 boot 2024-03-26T21:21:12.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T21:21:12.338 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[39202]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: purged_snaps scrub starts 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: purged_snaps scrub ok 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: osd.7 v1:172.21.15.138:6812/1004436505 boot 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:12.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:12 smithi088 ceph-mon[30075]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T21:21:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: purged_snaps scrub starts 2024-03-26T21:21:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: purged_snaps scrub ok 2024-03-26T21:21:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:21:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: osd.7 v1:172.21.15.138:6812/1004436505 boot 2024-03-26T21:21:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T21:21:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:21:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:12 smithi138 ceph-mon[33706]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T21:21:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.564+0000 7ff34ed83700 1 -- 172.21.15.88:0/2527107366 >> v1:172.21.15.88:6789/0 conn(0x7ff34810a570 legacy=0x7ff34810ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:12.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.564+0000 7ff34ed83700 1 -- 172.21.15.88:0/2527107366 shutdown_connections 2024-03-26T21:21:12.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.564+0000 7ff34ed83700 1 -- 172.21.15.88:0/2527107366 >> 172.21.15.88:0/2527107366 conn(0x7ff3480fc060 msgr2=0x7ff3480fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:12.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.564+0000 7ff34ed83700 1 -- 172.21.15.88:0/2527107366 shutdown_connections 2024-03-26T21:21:12.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.564+0000 7ff34ed83700 1 -- 172.21.15.88:0/2527107366 wait complete. 2024-03-26T21:21:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff34ed83700 1 Processor -- start 2024-03-26T21:21:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff34ed83700 1 -- start start 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff34ed83700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff348109e40 con 0x7ff348106910 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff34ed83700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3481a5d90 con 0x7ff34810a570 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff34ed83700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3481a6f70 con 0x7ff348100700 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff34cb1f700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff348100700 0x7ff3481092d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:32852/0 (socket says 172.21.15.88:32852) 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff347fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff348106910 0x7ff34819ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54314/0 (socket says 172.21.15.88:54314) 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff34cb1f700 1 -- 172.21.15.88:0/2665147027 learned_addr learned my addr 172.21.15.88:0/2665147027 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2425770048 0 0) 0x7ff348109e40 con 0x7ff348106910 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3380050b0 con 0x7ff348106910 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806512803 0 0) 0x7ff3481a6f70 con 0x7ff348100700 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff348109e40 con 0x7ff348100700 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3929830920 0 0) 0x7ff3481a5d90 con 0x7ff34810a570 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3481a6f70 con 0x7ff34810a570 2024-03-26T21:21:12.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3297878520 0 0) 0x7ff3380050b0 con 0x7ff348106910 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3481a5d90 con 0x7ff348106910 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff334001b10 con 0x7ff348106910 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1286246400 0 0) 0x7ff348109e40 con 0x7ff348100700 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3380050b0 con 0x7ff348100700 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff33c001830 con 0x7ff348100700 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2418496737 0 0) 0x7ff3481a6f70 con 0x7ff34810a570 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.565+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff348109e40 con 0x7ff34810a570 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff340001b80 con 0x7ff34810a570 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 149376405 0 0) 0x7ff3481a5d90 con 0x7ff348106910 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 >> v1:172.21.15.88:6790/0 conn(0x7ff348100700 legacy=0x7ff3481092d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 >> v1:172.21.15.138:6789/0 conn(0x7ff34810a570 legacy=0x7ff3481a3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3481a8190 con 0x7ff348106910 2024-03-26T21:21:12.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff3481a71a0 con 0x7ff348106910 2024-03-26T21:21:12.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7ff334003b10 con 0x7ff348106910 2024-03-26T21:21:12.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff334005cf0 con 0x7ff348106910 2024-03-26T21:21:12.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.566+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff3481a76e0 con 0x7ff348106910 2024-03-26T21:21:12.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.567+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff348068f30 con 0x7ff348106910 2024-03-26T21:21:12.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.567+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7ff334003680 con 0x7ff348106910 2024-03-26T21:21:12.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.568+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(55..55 src has 1..55) ==== 4110+0+0 (unknown 1948353659 0 0) 0x7ff3340891c0 con 0x7ff348106910 2024-03-26T21:21:12.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.573+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7ff334056ee0 con 0x7ff348106910 2024-03-26T21:21:12.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.775+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7ff348110ae0 con 0x7ff348106910 2024-03-26T21:21:12.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.776+0000 7ff345ffb700 1 -- 172.21.15.88:0/2665147027 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v56) ==== 74+0+130 (unknown 401554801 0 353482165) 0x7ff3340567e0 con 0x7ff348106910 2024-03-26T21:21:12.770 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:21:12.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.778+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 >> v1:172.21.15.88:6800/4097082509 conn(0x7ff33806f4b0 legacy=0x7ff338071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:12.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.778+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 >> v1:172.21.15.88:6789/0 conn(0x7ff348106910 legacy=0x7ff34819ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:12.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.778+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 shutdown_connections 2024-03-26T21:21:12.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.778+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 >> 172.21.15.88:0/2665147027 conn(0x7ff3480fc060 msgr2=0x7ff348104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:12.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.778+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 shutdown_connections 2024-03-26T21:21:12.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:12.778+0000 7ff34ed83700 1 -- 172.21.15.88:0/2665147027 wait complete. 2024-03-26T21:21:13.514 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1711488071,"num_in_osds":8,"osd_in_since":1711488053,"num_remapped_pgs":1} 2024-03-26T21:21:13.514 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:21:13.701 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T21:21:13.701 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2665147027' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T21:21:13.701 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:13.701 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T21:21:13.702 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2665147027' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:13.703 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:13 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.811 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:21:13.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2665147027' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:13.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:13 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:14 smithi138 ceph-mon[33706]: Detected new or changed devices on smithi138 2024-03-26T21:21:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:14 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi138 to 4272M 2024-03-26T21:21:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:14 smithi138 ceph-mon[33706]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T21:21:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:14 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:14.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[30075]: Detected new or changed devices on smithi138 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi138 to 4272M 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[30075]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[39202]: Detected new or changed devices on smithi138 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi138 to 4272M 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[39202]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:14 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:15.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.575+0000 7f6b198a3700 1 -- 172.21.15.88:0/1292320918 >> v1:172.21.15.88:6789/0 conn(0x7f6b1410b3b0 legacy=0x7f6b1410d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:15.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.575+0000 7f6b198a3700 1 -- 172.21.15.88:0/1292320918 shutdown_connections 2024-03-26T21:21:15.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.575+0000 7f6b198a3700 1 -- 172.21.15.88:0/1292320918 >> 172.21.15.88:0/1292320918 conn(0x7f6b140fcdd0 msgr2=0x7f6b140ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:15.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.575+0000 7f6b198a3700 1 -- 172.21.15.88:0/1292320918 shutdown_connections 2024-03-26T21:21:15.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.575+0000 7f6b198a3700 1 -- 172.21.15.88:0/1292320918 wait complete. 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.576+0000 7f6b198a3700 1 Processor -- start 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.576+0000 7f6b198a3700 1 -- start start 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b198a3700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b141a74e0 con 0x7f6b1410eec0 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b198a3700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b141a8700 con 0x7f6b141035a0 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b198a3700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b141a9940 con 0x7f6b1410b3b0 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b127fc700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f6b1410b3b0 0x7f6b141a22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:57076/0 (socket says 172.21.15.88:57076) 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b127fc700 1 -- 172.21.15.88:0/672404131 learned_addr learned my addr 172.21.15.88:0/672404131 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298078575 0 0) 0x7f6b141a74e0 con 0x7f6b1410eec0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6afc0031f0 con 0x7f6b1410eec0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878996986 0 0) 0x7f6b141a9940 con 0x7f6b1410b3b0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b141a74e0 con 0x7f6b1410b3b0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781009144 0 0) 0x7f6b141a8700 con 0x7f6b141035a0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b141a9940 con 0x7f6b141035a0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1506541139 0 0) 0x7f6afc0031f0 con 0x7f6b1410eec0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b141a8700 con 0x7f6b1410eec0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6b0c001610 con 0x7f6b1410eec0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1192309445 0 0) 0x7f6b141a74e0 con 0x7f6b1410b3b0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6afc0031f0 con 0x7f6b1410b3b0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6b08004520 con 0x7f6b1410b3b0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1523940242 0 0) 0x7f6b141a9940 con 0x7f6b141035a0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b141a74e0 con 0x7f6b141035a0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6b040017a0 con 0x7f6b141035a0 2024-03-26T21:21:15.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.577+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 233459832 0 0) 0x7f6b141a8700 con 0x7f6b1410eec0 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 >> v1:172.21.15.88:6790/0 conn(0x7f6b1410b3b0 legacy=0x7f6b141a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 >> v1:172.21.15.138:6789/0 conn(0x7f6b141035a0 legacy=0x7f6b1410abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b141aab80 con 0x7f6b1410eec0 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b141a9b70 con 0x7f6b1410eec0 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b141aa030 con 0x7f6b1410eec0 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f6b0c003e90 con 0x7f6b1410eec0 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6b0c004db0 con 0x7f6b1410eec0 2024-03-26T21:21:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.578+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b00005120 con 0x7f6b1410eec0 2024-03-26T21:21:15.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.579+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7f6b0c01b950 con 0x7f6b1410eec0 2024-03-26T21:21:15.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.579+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (unknown 3497627193 0 0) 0x7f6b0c088390 con 0x7f6b1410eec0 2024-03-26T21:21:15.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.584+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f6b0c0560c0 con 0x7f6b1410eec0 2024-03-26T21:21:15.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.786+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b00005410 con 0x7f6b1410eec0 2024-03-26T21:21:15.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.787+0000 7f6b188a1700 1 -- 172.21.15.88:0/672404131 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+11496 (unknown 2626357731 0 892875537) 0x7f6b0c0559c0 con 0x7f6b1410eec0 2024-03-26T21:21:15.780 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:21:15.780 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":58,"fsid":"e240a1a4-ebb5-11ee-95d0-87774f69a715","created":"2024-03-26T21:15:46.786114+0000","modified":"2024-03-26T21:21:14.706020+0000","last_up_change":"2024-03-26T21:21:11.241387+0000","last_in_change":"2024-03-26T21:20:53.506385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:19:03.150113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09199105-49c9-4d89-8c77-9193d4a78bf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":3845975585}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":3845975585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":3845975585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":3845975585}]},"public_addr":"172.21.15.88:6801/3845975585","cluster_addr":"172.21.15.88:6802/3845975585","heartbeat_back_addr":"172.21.15.88:6804/3845975585","heartbeat_front_addr":"172.21.15.88:6803/3845975585","state":["exists","up"]},{"osd":1,"uuid":"e2759e98-1df5-4e4d-a7f0-719202f0bc5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":698625447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":698625447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":698625447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":698625447}]},"public_addr":"172.21.15.88:6805/698625447","cluster_addr":"172.21.15.88:6806/698625447","heartbeat_back_addr":"172.21.15.88:6808/698625447","heartbeat_front_addr":"172.21.15.88:6807/698625447","state":["exists","up"]},{"osd":2,"uuid":"5bb8665c-3450-49dc-86fb-0d086ffb8ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":2517903952}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":2517903952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":2517903952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":2517903952}]},"public_addr":"172.21.15.88:6809/2517903952","cluster_addr":"172.21.15.88:6810/2517903952","heartbeat_back_addr":"172.21.15.88:6812/2517903952","heartbeat_front_addr":"172.21.15.88:6811/2517903952","state":["exists","up"]},{"osd":3,"uuid":"a727a6dd-71c5-4867-a2ad-91423368c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":3938069994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":3938069994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6816","nonce":3938069994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":3938069994}]},"public_addr":"172.21.15.88:6813/3938069994","cluster_addr":"172.21.15.88:6814/3938069994","heartbeat_back_addr":"172.21.15.88:6816/3938069994","heartbeat_front_addr":"172.21.15.88:6815/3938069994","state":["exists","up"]},{"osd":4,"uuid":"a2a562d8-4071-44c7-9695-7652271cbd52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6800","nonce":2187218446}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6801","nonce":2187218446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6803","nonce":2187218446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6802","nonce":2187218446}]},"public_addr":"172.21.15.138:6800/2187218446","cluster_addr":"172.21.15.138:6801/2187218446","heartbeat_back_addr":"172.21.15.138:6803/2187218446","heartbeat_front_addr":"172.21.15.138:6802/2187218446","state":["exists","up"]},{"osd":5,"uuid":"6c2c5207-7f0a-4700-b8e1-afeba8499378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6804","nonce":2329820906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6805","nonce":2329820906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6807","nonce":2329820906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6806","nonce":2329820906}]},"public_addr":"172.21.15.138:6804/2329820906","cluster_addr":"172.21.15.138:6805/2329820906","heartbeat_back_addr":"172.21.15.138:6807/2329820906","heartbeat_front_addr":"172.21.15.138:6806/2329820906","state":["exists","up"]},{"osd":6,"uuid":"2dbb6451-7d30-4039-9b3d-f01e9acd00c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6808","nonce":771295543}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6809","nonce":771295543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6811","nonce":771295543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6810","nonce":771295543}]},"public_addr":"172.21.15.138:6808/771295543","cluster_addr":"172.21.15.138:6809/771295543","heartbeat_back_addr":"172.21.15.138:6811/771295543","heartbeat_front_addr":"172.21.15.138:6810/771295543","state":["exists","up"]},{"osd":7,"uuid":"e0ba8a7d-0c56-4b23-838a-866cad1b91a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6812","nonce":1004436505}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6813","nonce":1004436505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6815","nonce":1004436505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6814","nonce":1004436505}]},"public_addr":"172.21.15.138:6812/1004436505","cluster_addr":"172.21.15.138:6813/1004436505","heartbeat_back_addr":"172.21.15.138:6815/1004436505","heartbeat_front_addr":"172.21.15.138:6814/1004436505","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:18:09.042877+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:18:34.576623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:00.195601+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:26.199781+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:50.484242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:20:17.097720+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:20:43.230683+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:21:10.028091+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6800/3808082170":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/2491754424":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/467543406":"2024-03-27T21:16:20.042822+0000","172.21.15.88:6800/221932617":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/2048865678":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/2427433423":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/1032838430":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/2073122482":"2024-03-27T21:16:47.065067+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:21:15.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.789+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 >> v1:172.21.15.88:6800/4097082509 conn(0x7f6afc06f4f0 legacy=0x7f6afc0719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:15.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.789+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 >> v1:172.21.15.88:6789/0 conn(0x7f6b1410eec0 legacy=0x7f6b141a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:15.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.789+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 shutdown_connections 2024-03-26T21:21:15.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.789+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 >> 172.21.15.88:0/672404131 conn(0x7f6b140fcdd0 msgr2=0x7f6b14111310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:15.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.789+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 shutdown_connections 2024-03-26T21:21:15.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:15.789+0000 7f6b198a3700 1 -- 172.21.15.88:0/672404131 wait complete. 2024-03-26T21:21:15.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:15 smithi138 ceph-mon[33706]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T21:21:15.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:15 smithi138 ceph-mon[33706]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:15.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:15 smithi088 ceph-mon[30075]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T21:21:15.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:15 smithi088 ceph-mon[30075]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:15.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:15 smithi088 ceph-mon[39202]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T21:21:15.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:15 smithi088 ceph-mon[39202]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:16.407 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T21:19:03.150113+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T21:21:16.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T21:21:16.645 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:21:16.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/672404131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:21:16.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/672404131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:21:16.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/672404131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:21:17.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:17 smithi138 ceph-mon[33706]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:17.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:17 smithi088 ceph-mon[30075]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:17.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:17 smithi088 ceph-mon[39202]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:21:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.309+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/2201994592 >> v1:172.21.15.88:6790/0 conn(0x7f9e4810a260 legacy=0x7f9e4810c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.309+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/2201994592 shutdown_connections 2024-03-26T21:21:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.309+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/2201994592 >> 172.21.15.88:0/2201994592 conn(0x7f9e480765d0 msgr2=0x7f9e480769e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.309+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/2201994592 shutdown_connections 2024-03-26T21:21:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.309+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/2201994592 wait complete. 2024-03-26T21:21:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.309+0000 7f9e4e9fb700 1 Processor -- start 2024-03-26T21:21:18.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.309+0000 7f9e4e9fb700 1 -- start start 2024-03-26T21:21:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e4e9fb700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9e481a6360 con 0x7f9e48101f80 2024-03-26T21:21:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e4e9fb700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9e481a7580 con 0x7f9e4810de00 2024-03-26T21:21:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e4e9fb700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9e481a87c0 con 0x7f9e4810a260 2024-03-26T21:21:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e47fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9e48101f80 0x7f9e48106700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34734/0 (socket says 172.21.15.88:34734) 2024-03-26T21:21:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e47fff700 1 -- 172.21.15.88:0/1869692651 learned_addr learned my addr 172.21.15.88:0/1869692651 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:21:18.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e477fe700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9e4810a260 0x7f9e481a1270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:57094/0 (socket says 172.21.15.88:57094) 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660706358 0 0) 0x7f9e481a87c0 con 0x7f9e4810a260 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e300035b0 con 0x7f9e4810a260 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974074554 0 0) 0x7f9e481a6360 con 0x7f9e48101f80 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e481a87c0 con 0x7f9e48101f80 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3616603936 0 0) 0x7f9e481a7580 con 0x7f9e4810de00 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e481a6360 con 0x7f9e4810de00 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3335261052 0 0) 0x7f9e481a87c0 con 0x7f9e48101f80 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e481a7580 con 0x7f9e48101f80 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1017201757 0 0) 0x7f9e300035b0 con 0x7f9e4810a260 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e481a87c0 con 0x7f9e4810a260 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9e3c0044a0 con 0x7f9e4810a260 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.310+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9e38001a30 con 0x7f9e48101f80 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3514668362 0 0) 0x7f9e481a6360 con 0x7f9e4810de00 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e300035b0 con 0x7f9e4810de00 2024-03-26T21:21:18.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3390917697 0 0) 0x7f9e481a7580 con 0x7f9e48101f80 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 >> v1:172.21.15.88:6790/0 conn(0x7f9e4810a260 legacy=0x7f9e481a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 >> v1:172.21.15.138:6789/0 conn(0x7f9e4810de00 legacy=0x7f9e481a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e481a9a00 con 0x7f9e48101f80 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9e481a6590 con 0x7f9e48101f80 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9e481a6a50 con 0x7f9e48101f80 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f9e38001ca0 con 0x7f9e48101f80 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.311+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9e38005c80 con 0x7f9e48101f80 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.312+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e34005120 con 0x7f9e48101f80 2024-03-26T21:21:18.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.312+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7f9e38003930 con 0x7f9e48101f80 2024-03-26T21:21:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.313+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (unknown 3497627193 0 0) 0x7f9e38089490 con 0x7f9e48101f80 2024-03-26T21:21:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.317+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f9e380571e0 con 0x7f9e48101f80 2024-03-26T21:21:18.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.512+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e34005c20 con 0x7f9e48101f80 2024-03-26T21:21:18.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.513+0000 7f9e457fa700 1 -- 172.21.15.88:0/1869692651 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (unknown 3844627101 0 2170607528) 0x7f9e38056ae0 con 0x7f9e48101f80 2024-03-26T21:21:18.507 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2024-03-26T21:21:18.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.515+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 >> v1:172.21.15.88:6800/4097082509 conn(0x7f9e300716e0 legacy=0x7f9e30073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:18.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.515+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 >> v1:172.21.15.88:6789/0 conn(0x7f9e48101f80 legacy=0x7f9e48106700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:18.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.515+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 shutdown_connections 2024-03-26T21:21:18.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.515+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 >> 172.21.15.88:0/1869692651 conn(0x7f9e480765d0 msgr2=0x7f9e480ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:18.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.515+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 shutdown_connections 2024-03-26T21:21:18.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:18.515+0000 7f9e4e9fb700 1 -- 172.21.15.88:0/1869692651 wait complete. 2024-03-26T21:21:18.818 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1869692651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T21:21:18.818 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1869692651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T21:21:18.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1869692651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T21:21:19.154 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi088 2024-03-26T21:21:19.155 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi088=foo.a' 2024-03-26T21:21:19.404 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:19.794 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:19 smithi138 ceph-mon[33706]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-26T21:21:19.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:19 smithi088 ceph-mon[30075]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-26T21:21:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:19 smithi088 ceph-mon[39202]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-26T21:21:21.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.196+0000 7fc906cda700 1 -- 172.21.15.138:0/3555058004 >> v1:172.21.15.138:6789/0 conn(0x7fc900101330 legacy=0x7fc900101750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:21.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.196+0000 7fc906cda700 1 -- 172.21.15.138:0/3555058004 shutdown_connections 2024-03-26T21:21:21.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.196+0000 7fc906cda700 1 -- 172.21.15.138:0/3555058004 >> 172.21.15.138:0/3555058004 conn(0x7fc9000fc8d0 msgr2=0x7fc9000fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:21.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.196+0000 7fc906cda700 1 -- 172.21.15.138:0/3555058004 shutdown_connections 2024-03-26T21:21:21.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.196+0000 7fc906cda700 1 -- 172.21.15.138:0/3555058004 wait complete. 2024-03-26T21:21:21.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.196+0000 7fc906cda700 1 Processor -- start 2024-03-26T21:21:21.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc906cda700 1 -- start start 2024-03-26T21:21:21.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc906cda700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc900197460 con 0x7fc900101330 2024-03-26T21:21:21.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc906cda700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9001a2c50 con 0x7fc900105760 2024-03-26T21:21:21.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc906cda700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9001a3e30 con 0x7fc9001094a0 2024-03-26T21:21:21.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f7fff700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc900105760 0x7fc90019cdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:52094/0 (socket says 172.21.15.138:52094) 2024-03-26T21:21:21.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f7fff700 1 -- 172.21.15.138:0/430295368 learned_addr learned my addr 172.21.15.138:0/430295368 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3898142497 0 0) 0x7fc9001a2c50 con 0x7fc900105760 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e80035b0 con 0x7fc900105760 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2387268009 0 0) 0x7fc9001a3e30 con 0x7fc9001094a0 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9001a2c50 con 0x7fc9001094a0 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3310632917 0 0) 0x7fc900197460 con 0x7fc900101330 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9001a3e30 con 0x7fc900101330 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1777775491 0 0) 0x7fc8e80035b0 con 0x7fc900105760 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc900197460 con 0x7fc900105760 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc8f00018b0 con 0x7fc900105760 2024-03-26T21:21:21.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 658301710 0 0) 0x7fc9001a2c50 con 0x7fc9001094a0 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8e80035b0 con 0x7fc9001094a0 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc8f8001b70 con 0x7fc9001094a0 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 634295705 0 0) 0x7fc900197460 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 >> v1:172.21.15.88:6790/0 conn(0x7fc9001094a0 legacy=0x7fc9001a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.197+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 >> v1:172.21.15.88:6789/0 conn(0x7fc900101330 legacy=0x7fc900196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.198+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9001a5050 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.198+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9001a4060 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.198+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fc8f0001ca0 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.198+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fc8f0005cc0 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.198+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9001a45a0 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.199+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc900068f30 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.199+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7fc8f0003950 con 0x7fc900105760 2024-03-26T21:21:21.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.200+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (unknown 3497627193 0 0) 0x7fc8f0089410 con 0x7fc900105760 2024-03-26T21:21:21.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.204+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fc8f0057060 con 0x7fc900105760 2024-03-26T21:21:21.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.414+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi088=foo.a", "target": ["mon-mgr", ""]}) -- 0x7fc9001a4930 con 0x7fc8e8071460 2024-03-26T21:21:21.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.423+0000 7fc8f5ffb700 1 -- 172.21.15.138:0/430295368 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (unknown 0 0 1123153589) 0x7fc9001a4930 con 0x7fc8e8071460 2024-03-26T21:21:21.416 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled rgw.foo.a update... 2024-03-26T21:21:21.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.425+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 >> v1:172.21.15.88:6800/4097082509 conn(0x7fc8e8071460 legacy=0x7fc8e8073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:21.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.425+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 >> v1:172.21.15.138:6789/0 conn(0x7fc900105760 legacy=0x7fc90019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:21.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.425+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 shutdown_connections 2024-03-26T21:21:21.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.425+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 >> 172.21.15.138:0/430295368 conn(0x7fc9000fc8d0 msgr2=0x7fc900107bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:21.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.425+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 shutdown_connections 2024-03-26T21:21:21.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:21.425+0000 7fc906cda700 1 -- 172.21.15.138:0/430295368 wait complete. 2024-03-26T21:21:22.022 DEBUG:teuthology.orchestra.run.smithi088:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T21:21:22.024 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi138 2024-03-26T21:21:22.024 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T21:21:22.263 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:22.373 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:21:22.373 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:22.374 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:22 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:22.384 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T21:21:22.384 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.384 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:22.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:22.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:22.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T21:21:22.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T21:21:22.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:22.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:22 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:23.367 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:23 smithi138 ceph-mon[33706]: from='client.24352 v1:172.21.15.138:0/430295368' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi088=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:23.367 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:23 smithi138 ceph-mon[33706]: Saving service rgw.foo.a spec with placement smithi088=foo.a;count:1 2024-03-26T21:21:23.367 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:23 smithi138 ceph-mon[33706]: Deploying daemon rgw.foo.a on smithi088 2024-03-26T21:21:23.474 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:23 smithi088 ceph-mon[39202]: from='client.24352 v1:172.21.15.138:0/430295368' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi088=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:23.474 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:23 smithi088 ceph-mon[39202]: Saving service rgw.foo.a spec with placement smithi088=foo.a;count:1 2024-03-26T21:21:23.475 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:23 smithi088 ceph-mon[39202]: Deploying daemon rgw.foo.a on smithi088 2024-03-26T21:21:23.475 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:23 smithi088 ceph-mon[30075]: from='client.24352 v1:172.21.15.138:0/430295368' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi088=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:23.475 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:23 smithi088 ceph-mon[30075]: Saving service rgw.foo.a spec with placement smithi088=foo.a;count:1 2024-03-26T21:21:23.475 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:23 smithi088 ceph-mon[30075]: Deploying daemon rgw.foo.a on smithi088 2024-03-26T21:21:23.734 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:21:23 smithi088 systemd[1]: Starting Ceph rgw.foo.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:21:23.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.925+0000 7f65d064f700 1 -- 172.21.15.138:0/3015839203 >> v1:172.21.15.138:6789/0 conn(0x7f65c8106880 legacy=0x7f65c8108cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:23.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.925+0000 7f65d064f700 1 -- 172.21.15.138:0/3015839203 shutdown_connections 2024-03-26T21:21:23.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.925+0000 7f65d064f700 1 -- 172.21.15.138:0/3015839203 >> 172.21.15.138:0/3015839203 conn(0x7f65c80fd9f0 msgr2=0x7f65c80ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:23.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.925+0000 7f65d064f700 1 -- 172.21.15.138:0/3015839203 shutdown_connections 2024-03-26T21:21:23.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.925+0000 7f65d064f700 1 -- 172.21.15.138:0/3015839203 wait complete. 2024-03-26T21:21:23.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.925+0000 7f65d064f700 1 Processor -- start 2024-03-26T21:21:23.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65d064f700 1 -- start start 2024-03-26T21:21:23.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65d064f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65c8198200 con 0x7f65c810a5c0 2024-03-26T21:21:23.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65d064f700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65c81a39f0 con 0x7f65c8102450 2024-03-26T21:21:23.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65d064f700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65c81a4bd0 con 0x7f65c8106880 2024-03-26T21:21:23.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65ce3eb700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f65c8102450 0x7f65c81976b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38042/0 (socket says 172.21.15.138:38042) 2024-03-26T21:21:23.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65ce3eb700 1 -- 172.21.15.138:0/868466560 learned_addr learned my addr 172.21.15.138:0/868466560 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370026719 0 0) 0x7f65c81a39f0 con 0x7f65c8102450 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65b40035b0 con 0x7f65c8102450 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3748751461 0 0) 0x7f65c8198200 con 0x7f65c810a5c0 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65c81a39f0 con 0x7f65c810a5c0 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4099787167 0 0) 0x7f65c81a4bd0 con 0x7f65c8106880 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65c8198200 con 0x7f65c8106880 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1644499905 0 0) 0x7f65b40035b0 con 0x7f65c8102450 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65c81a4bd0 con 0x7f65c8102450 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f65c4001980 con 0x7f65c8102450 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2543550134 0 0) 0x7f65c81a39f0 con 0x7f65c810a5c0 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65b40035b0 con 0x7f65c810a5c0 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1618986918 0 0) 0x7f65c8198200 con 0x7f65c8106880 2024-03-26T21:21:23.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65c81a39f0 con 0x7f65c8106880 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f65c0001a60 con 0x7f65c810a5c0 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f65b8004550 con 0x7f65c8106880 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.926+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 534341512 0 0) 0x7f65c81a4bd0 con 0x7f65c8102450 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.927+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 >> v1:172.21.15.88:6790/0 conn(0x7f65c8106880 legacy=0x7f65c819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.927+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 >> v1:172.21.15.88:6789/0 conn(0x7f65c810a5c0 legacy=0x7f65c81a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.927+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c81a5df0 con 0x7f65c8102450 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.927+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65c81a2a40 con 0x7f65c8102450 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.927+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65c81a2f80 con 0x7f65c8102450 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.927+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f65c4003de0 con 0x7f65c8102450 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.927+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f65c4005dd0 con 0x7f65c8102450 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.928+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ac005150 con 0x7f65c8102450 2024-03-26T21:21:23.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.928+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7f65c4003950 con 0x7f65c8102450 2024-03-26T21:21:23.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.929+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (unknown 3497627193 0 0) 0x7f65c4089460 con 0x7f65c8102450 2024-03-26T21:21:23.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:23.933+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f65c4005dd0 con 0x7f65c8102450 2024-03-26T21:21:24.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.128+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7f65ac005440 con 0x7f65c8102450 2024-03-26T21:21:24.157 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:21:23 smithi088 podman[67285]: 2024-03-26 21:21:23.862161607 +0000 UTC m=+0.111538741 container create 2d95ae11981cabc1abf725e0aa6bb26b240e295b3e76458dd98ffe04f3bb52c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T21:21:24.157 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:21:23 smithi088 podman[67285]: 2024-03-26 21:21:23.768300302 +0000 UTC m=+0.017677390 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:21:24.157 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:21:24 smithi088 podman[67285]: 2024-03-26 21:21:24.056227605 +0000 UTC m=+0.305604693 container init 2d95ae11981cabc1abf725e0aa6bb26b240e295b3e76458dd98ffe04f3bb52c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T21:21:24.157 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:21:24 smithi088 podman[67285]: 2024-03-26 21:21:24.061503002 +0000 UTC m=+0.310880090 container start 2d95ae11981cabc1abf725e0aa6bb26b240e295b3e76458dd98ffe04f3bb52c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-26T21:21:24.158 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:21:24 smithi088 bash[67285]: 2d95ae11981cabc1abf725e0aa6bb26b240e295b3e76458dd98ffe04f3bb52c4 2024-03-26T21:21:24.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.181+0000 7f65bf7fe700 1 -- 172.21.15.138:0/868466560 <== mon.1 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) ==== 160+0+0 (unknown 1751278495 0 0) 0x7f65c4057190 con 0x7f65c8102450 2024-03-26T21:21:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.185+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 >> v1:172.21.15.88:6800/4097082509 conn(0x7f65b406f4b0 legacy=0x7f65b4071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.185+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 >> v1:172.21.15.138:6789/0 conn(0x7f65c8102450 legacy=0x7f65c81976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.185+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 shutdown_connections 2024-03-26T21:21:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.185+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 >> 172.21.15.138:0/868466560 conn(0x7f65c80fd9f0 msgr2=0x7f65c8109d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.185+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 shutdown_connections 2024-03-26T21:21:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:24.185+0000 7f65d064f700 1 -- 172.21.15.138:0/868466560 wait complete. 2024-03-26T21:21:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'datapool' created 2024-03-26T21:21:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:24 smithi138 ceph-mon[33706]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-26T21:21:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/868466560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:21:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:24 smithi138 ceph-mon[33706]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:21:24.483 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:21:24 smithi088 systemd[1]: Started Ceph rgw.foo.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:21:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:24 smithi088 ceph-mon[30075]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-26T21:21:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/868466560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:21:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:24 smithi088 ceph-mon[30075]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:21:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:24 smithi088 ceph-mon[39202]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-26T21:21:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/868466560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:21:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:24 smithi088 ceph-mon[39202]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:21:24.809 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T21:21:25.043 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:25.461 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T21:21:25.461 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T21:21:25.461 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3947657233' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: Saving service rgw.foo.a spec with placement smithi088=foo.a;count:1 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3947657233' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T21:21:25.462 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:25 smithi138 ceph-mon[33706]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T21:21:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T21:21:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T21:21:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3947657233' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T21:21:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: Saving service rgw.foo.a spec with placement smithi088=foo.a;count:1 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3947657233' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[39202]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3947657233' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: Saving service rgw.foo.a spec with placement smithi088=foo.a;count:1 2024-03-26T21:21:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:25.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:25.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3947657233' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T21:21:25.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:25 smithi088 ceph-mon[30075]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T21:21:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: pgmap v188: 36 pgs: 4 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:21:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:21:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T21:21:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:21:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:21:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: pgmap v188: 36 pgs: 4 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:21:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:21:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: pgmap v188: 36 pgs: 4 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T21:21:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:21:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:21:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: Checking dashboard <-> RGW credentials 2024-03-26T21:21:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/528065627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:21:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: from='client.24412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:21:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:21:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:21:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: from='client.24412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T21:21:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:27 smithi138 ceph-mon[33706]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T21:21:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: Checking dashboard <-> RGW credentials 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/528065627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: from='client.24412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: from='client.24412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[39202]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: Checking dashboard <-> RGW credentials 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/528065627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: from='client.24412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:21:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:21:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: from='client.24412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T21:21:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:27 smithi088 ceph-mon[30075]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T21:21:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:28 smithi138 ceph-mon[33706]: pgmap v191: 68 pgs: 2 creating+activating, 18 creating+peering, 8 active+clean, 40 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T21:21:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:28 smithi138 ceph-mon[33706]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T21:21:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:21:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:21:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[39202]: pgmap v191: 68 pgs: 2 creating+activating, 18 creating+peering, 8 active+clean, 40 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T21:21:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[39202]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T21:21:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:21:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:21:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[30075]: pgmap v191: 68 pgs: 2 creating+activating, 18 creating+peering, 8 active+clean, 40 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T21:21:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[30075]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T21:21:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:21:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:21:29.877 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.138 --placement '1;smithi138=iscsi.a' 2024-03-26T21:21:30.116 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:30 smithi138 ceph-mon[33706]: pgmap v194: 100 pgs: 2 creating+activating, 28 creating+peering, 33 active+clean, 37 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:21:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:21:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:21:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:30 smithi138 ceph-mon[33706]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T21:21:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[39202]: pgmap v194: 100 pgs: 2 creating+activating, 28 creating+peering, 33 active+clean, 37 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:21:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:21:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:21:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[39202]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T21:21:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[30075]: pgmap v194: 100 pgs: 2 creating+activating, 28 creating+peering, 33 active+clean, 37 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:21:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:21:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:21:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:30 smithi088 ceph-mon[30075]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T21:21:31.321 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:31 smithi138 ceph-mon[33706]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T21:21:31.321 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:21:31.322 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:21:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:31 smithi088 ceph-mon[39202]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T21:21:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:21:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:21:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:31 smithi088 ceph-mon[30075]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T21:21:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:21:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:21:31.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.868+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/2724898202 >> v1:172.21.15.138:6789/0 conn(0x7f3ae8102f40 legacy=0x7f3ae8105330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:31.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.868+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/2724898202 shutdown_connections 2024-03-26T21:21:31.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.868+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/2724898202 >> 172.21.15.138:0/2724898202 conn(0x7f3ae80fc8d0 msgr2=0x7f3ae80fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:31.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.868+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/2724898202 shutdown_connections 2024-03-26T21:21:31.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.868+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/2724898202 wait complete. 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.868+0000 7f3aeeaf1700 1 Processor -- start 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.868+0000 7f3aeeaf1700 1 -- start start 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3aeeaf1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ae81986e0 con 0x7f3ae8106a50 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3aeeaf1700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ae81a3ed0 con 0x7f3ae8102f40 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3aeeaf1700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ae81a50b0 con 0x7f3ae810a560 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3aec88d700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f3ae8102f40 0x7f3ae8197b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:51248/0 (socket says 172.21.15.138:51248) 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3aec88d700 1 -- 172.21.15.138:0/3110603074 learned_addr learned my addr 172.21.15.138:0/3110603074 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050083725 0 0) 0x7f3ae81a3ed0 con 0x7f3ae8102f40 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ad00035b0 con 0x7f3ae8102f40 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084834051 0 0) 0x7f3ae81986e0 con 0x7f3ae8106a50 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae81a3ed0 con 0x7f3ae8106a50 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899763381 0 0) 0x7f3ae81a50b0 con 0x7f3ae810a560 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae81986e0 con 0x7f3ae810a560 2024-03-26T21:21:31.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3004594954 0 0) 0x7f3ad00035b0 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ae81a50b0 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3ae40017b0 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 125277382 0 0) 0x7f3ae81a3ed0 con 0x7f3ae8106a50 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ad00035b0 con 0x7f3ae8106a50 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3ad8001830 con 0x7f3ae8106a50 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3260694588 0 0) 0x7f3ae81a50b0 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 >> v1:172.21.15.88:6790/0 conn(0x7f3ae810a560 legacy=0x7f3ae81a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.869+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 >> v1:172.21.15.88:6789/0 conn(0x7f3ae8106a50 legacy=0x7f3ae819e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.870+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae81a62d0 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.870+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae81a2f20 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.870+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f3ae4001f30 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.870+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3ae40055a0 con 0x7f3ae8102f40 2024-03-26T21:21:31.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.870+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ae81a3460 con 0x7f3ae8102f40 2024-03-26T21:21:31.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.871+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae810ff90 con 0x7f3ae8102f40 2024-03-26T21:21:31.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.871+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7f3ae40055c0 con 0x7f3ae8102f40 2024-03-26T21:21:31.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.872+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(66..66 src has 1..66) ==== 5926+0+0 (unknown 2441340603 0 0) 0x7f3ae408a190 con 0x7f3ae8102f40 2024-03-26T21:21:31.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:31.876+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f3ae4057790 con 0x7f3ae8102f40 2024-03-26T21:21:32.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.092+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7f3ae81a3750 con 0x7f3ad006f200 2024-03-26T21:21:32.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.100+0000 7f3addffb700 1 -- 172.21.15.138:0/3110603074 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (unknown 0 0 803663096) 0x7f3ae81a3750 con 0x7f3ad006f200 2024-03-26T21:21:32.094 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled iscsi.datapool update... 2024-03-26T21:21:32.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.102+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 >> v1:172.21.15.88:6800/4097082509 conn(0x7f3ad006f200 legacy=0x7f3ad00716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:32.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.102+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 >> v1:172.21.15.138:6789/0 conn(0x7f3ae8102f40 legacy=0x7f3ae8197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:32.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.102+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 shutdown_connections 2024-03-26T21:21:32.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.102+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 >> 172.21.15.138:0/3110603074 conn(0x7f3ae80fc8d0 msgr2=0x7f3ae80fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:32.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.102+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 shutdown_connections 2024-03-26T21:21:32.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:32.102+0000 7f3aeeaf1700 1 -- 172.21.15.138:0/3110603074 wait complete. 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: pgmap v197: 132 pgs: 24 creating+peering, 56 active+clean, 52 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:21:32.471 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:32 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:32.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: pgmap v197: 132 pgs: 24 creating+peering, 56 active+clean, 52 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T21:21:32.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: pgmap v197: 132 pgs: 24 creating+peering, 56 active+clean, 52 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T21:21:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:21:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:21:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:21:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:32 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:32.801 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T21:21:32.801 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:21:32.801 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T21:21:32.832 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:21:32.832 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T21:21:32.862 DEBUG:teuthology.orchestra.run.smithi138:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T21:21:32.904 INFO:tasks.cephadm:Adding prometheus.a on smithi138 2024-03-26T21:21:32.905 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi138=a' 2024-03-26T21:21:33.342 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='client.24418 v1:172.21.15.138:0/3110603074' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: Saving service iscsi.datapool spec with placement smithi138=iscsi.a;count:1 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T21:21:33.343 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:33 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='client.24418 v1:172.21.15.138:0/3110603074' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: Saving service iscsi.datapool spec with placement smithi138=iscsi.a;count:1 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T21:21:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='client.24418 v1:172.21.15.138:0/3110603074' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: Saving service iscsi.datapool spec with placement smithi138=iscsi.a;count:1 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3159689879' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2049445430' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T21:21:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:33 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:34.323 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:34 smithi138 ceph-mon[33706]: Deploying daemon iscsi.iscsi.a on smithi138 2024-03-26T21:21:34.323 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:34 smithi138 ceph-mon[33706]: pgmap v200: 132 pgs: 5 creating+peering, 109 active+clean, 18 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:21:34.324 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:34 smithi138 ceph-mon[33706]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:21:34.324 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:34 smithi138 ceph-mon[33706]: Cluster is now healthy 2024-03-26T21:21:34.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[30075]: Deploying daemon iscsi.iscsi.a on smithi138 2024-03-26T21:21:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[30075]: pgmap v200: 132 pgs: 5 creating+peering, 109 active+clean, 18 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:21:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[30075]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:21:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[30075]: Cluster is now healthy 2024-03-26T21:21:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[39202]: Deploying daemon iscsi.iscsi.a on smithi138 2024-03-26T21:21:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[39202]: pgmap v200: 132 pgs: 5 creating+peering, 109 active+clean, 18 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:21:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[39202]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:21:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:34 smithi088 ceph-mon[39202]: Cluster is now healthy 2024-03-26T21:21:35.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.963+0000 7fd7100fe700 1 -- 172.21.15.138:0/585510159 >> v1:172.21.15.138:6789/0 conn(0x7fd708105760 legacy=0x7fd708107bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:35.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.963+0000 7fd7100fe700 1 -- 172.21.15.138:0/585510159 shutdown_connections 2024-03-26T21:21:35.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.963+0000 7fd7100fe700 1 -- 172.21.15.138:0/585510159 >> 172.21.15.138:0/585510159 conn(0x7fd7080fc8d0 msgr2=0x7fd7080fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:35.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.963+0000 7fd7100fe700 1 -- 172.21.15.138:0/585510159 shutdown_connections 2024-03-26T21:21:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.963+0000 7fd7100fe700 1 -- 172.21.15.138:0/585510159 wait complete. 2024-03-26T21:21:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd7100fe700 1 Processor -- start 2024-03-26T21:21:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd7100fe700 1 -- start start 2024-03-26T21:21:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd7100fe700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd708197450 con 0x7fd7081094a0 2024-03-26T21:21:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd7100fe700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd7081a2c40 con 0x7fd708101330 2024-03-26T21:21:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd7100fe700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd7081a3e20 con 0x7fd708105760 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd70de9a700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fd708101330 0x7fd708196900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:51260/0 (socket says 172.21.15.138:51260) 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd70de9a700 1 -- 172.21.15.138:0/1206098811 learned_addr learned my addr 172.21.15.138:0/1206098811 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1886735946 0 0) 0x7fd7081a2c40 con 0x7fd708101330 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f00035b0 con 0x7fd708101330 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200479919 0 0) 0x7fd708197450 con 0x7fd7081094a0 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7081a2c40 con 0x7fd7081094a0 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1299181208 0 0) 0x7fd7081a3e20 con 0x7fd708105760 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd708197450 con 0x7fd708105760 2024-03-26T21:21:35.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3847074679 0 0) 0x7fd6f00035b0 con 0x7fd708101330 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7081a3e20 con 0x7fd708101330 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.964+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd700001bf0 con 0x7fd708101330 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2314276586 0 0) 0x7fd7081a2c40 con 0x7fd7081094a0 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6f00035b0 con 0x7fd7081094a0 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1824229840 0 0) 0x7fd7081a3e20 con 0x7fd708101330 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 >> v1:172.21.15.88:6790/0 conn(0x7fd708105760 legacy=0x7fd70819cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 >> v1:172.21.15.88:6789/0 conn(0x7fd7081094a0 legacy=0x7fd7081a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7081a5040 con 0x7fd708101330 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7081a1c90 con 0x7fd708101330 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7081a21d0 con 0x7fd708101330 2024-03-26T21:21:35.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fd700001fc0 con 0x7fd708101330 2024-03-26T21:21:35.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.965+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd7000060b0 con 0x7fd708101330 2024-03-26T21:21:35.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.966+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f4005150 con 0x7fd708101330 2024-03-26T21:21:35.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.966+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 15) ==== 92439+0+0 (unknown 2877581041 0 0) 0x7fd7000043d0 con 0x7fd708101330 2024-03-26T21:21:35.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.967+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(67..67 src has 1..67) ==== 5942+0+0 (unknown 3136039053 0 0) 0x7fd700089f00 con 0x7fd708101330 2024-03-26T21:21:35.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:35.971+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fd7000573b0 con 0x7fd708101330 2024-03-26T21:21:36.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.186+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}) -- 0x7fd6f4000bc0 con 0x7fd6f006f210 2024-03-26T21:21:36.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.193+0000 7fd706ffd700 1 -- 172.21.15.138:0/1206098811 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (unknown 0 0 1342662408) 0x7fd6f4000bc0 con 0x7fd6f006f210 2024-03-26T21:21:36.187 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled prometheus update... 2024-03-26T21:21:36.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.195+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 >> v1:172.21.15.88:6800/4097082509 conn(0x7fd6f006f210 legacy=0x7fd6f00716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:36.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.195+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 >> v1:172.21.15.138:6789/0 conn(0x7fd708101330 legacy=0x7fd708196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:36.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.195+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 shutdown_connections 2024-03-26T21:21:36.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.195+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 >> 172.21.15.138:0/1206098811 conn(0x7fd7080fc8d0 msgr2=0x7fd708107bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:36.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.195+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 shutdown_connections 2024-03-26T21:21:36.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:36.195+0000 7fd7100fe700 1 -- 172.21.15.138:0/1206098811 wait complete. 2024-03-26T21:21:36.219 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:36 smithi138 ceph-mon[33706]: pgmap v201: 132 pgs: 132 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2.4 KiB/s wr, 58 op/s 2024-03-26T21:21:36.220 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:36 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:36 smithi088 ceph-mon[39202]: pgmap v201: 132 pgs: 132 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2.4 KiB/s wr, 58 op/s 2024-03-26T21:21:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:36 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:36 smithi088 ceph-mon[30075]: pgmap v201: 132 pgs: 132 active+clean; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2.4 KiB/s wr, 58 op/s 2024-03-26T21:21:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:36 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.204 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:21:36 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug Processing osd blocklist entries for this node 2024-03-26T21:21:37.266 DEBUG:teuthology.orchestra.run.smithi138:prometheus.a> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T21:21:37.269 INFO:tasks.cephadm:Adding node-exporter.a on smithi088 2024-03-26T21:21:37.269 INFO:tasks.cephadm:Adding node-exporter.b on smithi138 2024-03-26T21:21:37.269 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi088=a;smithi138=b' 2024-03-26T21:21:37.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug Reading the configuration object to update local LIO configuration 2024-03-26T21:21:37.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug Configuration does not have an entry for this host(smithi138.front.sepia.ceph.com) - nothing to define to LIO 2024-03-26T21:21:37.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-26T21:21:37.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: from='client.24439 v1:172.21.15.138:0/1206098811' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:37.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: Saving service prometheus spec with placement smithi138=a;count:1 2024-03-26T21:21:37.464 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T21:21:37.464 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.464 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.464 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.464 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.464 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:37.464 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:37 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: from='client.24439 v1:172.21.15.138:0/1206098811' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: Saving service prometheus spec with placement smithi138=a;count:1 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: from='client.24439 v1:172.21.15.138:0/1206098811' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: Saving service prometheus spec with placement smithi138=a;count:1 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:21:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:37 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:37.918 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:38.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:38 smithi138 ceph-mon[33706]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T21:21:38.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:38 smithi138 ceph-mon[33706]: pgmap v203: 132 pgs: 132 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 4.3 KiB/s wr, 151 op/s 2024-03-26T21:21:38.341 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/8106268' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T21:21:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:38 smithi088 ceph-mon[30075]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T21:21:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:38 smithi088 ceph-mon[30075]: pgmap v203: 132 pgs: 132 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 4.3 KiB/s wr, 151 op/s 2024-03-26T21:21:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/8106268' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T21:21:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:38 smithi088 ceph-mon[39202]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T21:21:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:38 smithi088 ceph-mon[39202]: pgmap v203: 132 pgs: 132 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 4.3 KiB/s wr, 151 op/s 2024-03-26T21:21:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/8106268' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.190+0000 7f405ad14700 1 -- 172.21.15.138:0/713037480 >> v1:172.21.15.138:6789/0 conn(0x7f4054106870 legacy=0x7f4054108cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.190+0000 7f405ad14700 1 -- 172.21.15.138:0/713037480 shutdown_connections 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.190+0000 7f405ad14700 1 -- 172.21.15.138:0/713037480 >> 172.21.15.138:0/713037480 conn(0x7f40540fd9e0 msgr2=0x7f40540ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.190+0000 7f405ad14700 1 -- 172.21.15.138:0/713037480 shutdown_connections 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.190+0000 7f405ad14700 1 -- 172.21.15.138:0/713037480 wait complete. 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.190+0000 7f405ad14700 1 Processor -- start 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f405ad14700 1 -- start start 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f405ad14700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4054198560 con 0x7f4054106870 2024-03-26T21:21:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f405ad14700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40541a3d50 con 0x7f405410a5b0 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f405ad14700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40541a4f30 con 0x7f4054102440 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f40592b1700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f405410a5b0 0x7f40541a1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:51346/0 (socket says 172.21.15.138:51346) 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f40592b1700 1 -- 172.21.15.138:0/2746422481 learned_addr learned my addr 172.21.15.138:0/2746422481 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719467310 0 0) 0x7f40541a3d50 con 0x7f405410a5b0 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40440035b0 con 0x7f405410a5b0 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268287712 0 0) 0x7f4054198560 con 0x7f4054106870 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40541a3d50 con 0x7f4054106870 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1392902490 0 0) 0x7f40541a4f30 con 0x7f4054102440 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4054198560 con 0x7f4054102440 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1392639590 0 0) 0x7f40440035b0 con 0x7f405410a5b0 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40541a4f30 con 0x7f405410a5b0 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f404c001c10 con 0x7f405410a5b0 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 495849120 0 0) 0x7f40541a3d50 con 0x7f4054106870 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40440035b0 con 0x7f4054106870 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4040001430 con 0x7f4054106870 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 430857363 0 0) 0x7f4054198560 con 0x7f4054102440 2024-03-26T21:21:40.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40541a3d50 con 0x7f4054102440 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f40480017f0 con 0x7f4054102440 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2361150541 0 0) 0x7f40541a4f30 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 >> v1:172.21.15.88:6790/0 conn(0x7f4054102440 legacy=0x7f4054197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 >> v1:172.21.15.88:6789/0 conn(0x7f4054106870 legacy=0x7f405419def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.191+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40541a6150 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.192+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40541a2da0 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.192+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f404c003eb0 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.192+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f404c005f30 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.192+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f40541a32e0 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.193+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4054068f30 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.193+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 16) ==== 92490+0+0 (unknown 196927556 0 0) 0x7f404c003a20 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.194+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(68..68 src has 1..68) ==== 5942+0+0 (unknown 1034119651 0 0) 0x7f404c08a030 con 0x7f405410a5b0 2024-03-26T21:21:40.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.198+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f404c057440 con 0x7f405410a5b0 2024-03-26T21:21:40.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.469+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi088=a;smithi138=b", "target": ["mon-mgr", ""]}) -- 0x7f40541a3640 con 0x7f404406f620 2024-03-26T21:21:40.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.475+0000 7f4051ffb700 1 -- 172.21.15.138:0/2746422481 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (unknown 0 0 240551134) 0x7f40541a3640 con 0x7f404406f620 2024-03-26T21:21:40.469 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled node-exporter update... 2024-03-26T21:21:40.469 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:40 smithi138 ceph-mon[33706]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 142 op/s 2024-03-26T21:21:40.469 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:40 smithi138 ceph-mon[33706]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T21:21:40.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.478+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 >> v1:172.21.15.88:6800/4097082509 conn(0x7f404406f620 legacy=0x7f4044071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:40.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.478+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 >> v1:172.21.15.138:6789/0 conn(0x7f405410a5b0 legacy=0x7f40541a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:40.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.478+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 shutdown_connections 2024-03-26T21:21:40.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.478+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 >> 172.21.15.138:0/2746422481 conn(0x7f40540fd9e0 msgr2=0x7f4054108cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:40.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.478+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 shutdown_connections 2024-03-26T21:21:40.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:40.478+0000 7f405ad14700 1 -- 172.21.15.138:0/2746422481 wait complete. 2024-03-26T21:21:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:40 smithi088 ceph-mon[39202]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 142 op/s 2024-03-26T21:21:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:40 smithi088 ceph-mon[39202]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T21:21:40.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:40 smithi088 ceph-mon[30075]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 142 op/s 2024-03-26T21:21:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:40 smithi088 ceph-mon[30075]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T21:21:41.090 DEBUG:teuthology.orchestra.run.smithi088:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T21:21:41.092 DEBUG:teuthology.orchestra.run.smithi138:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T21:21:41.095 INFO:tasks.cephadm:Adding alertmanager.a on smithi088 2024-03-26T21:21:41.095 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi088=a' 2024-03-26T21:21:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:41 smithi138 ceph-mon[33706]: from='client.24475 v1:172.21.15.138:0/2746422481' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi088=a;smithi138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:41 smithi138 ceph-mon[33706]: Saving service node-exporter spec with placement smithi088=a;smithi138=b;count:2 2024-03-26T21:21:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:41 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:41 smithi138 ceph-mon[33706]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 126 op/s 2024-03-26T21:21:41.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[30075]: from='client.24475 v1:172.21.15.138:0/2746422481' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi088=a;smithi138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:41.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[30075]: Saving service node-exporter spec with placement smithi088=a;smithi138=b;count:2 2024-03-26T21:21:41.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:41.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[30075]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 126 op/s 2024-03-26T21:21:41.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[39202]: from='client.24475 v1:172.21.15.138:0/2746422481' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi088=a;smithi138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:41.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[39202]: Saving service node-exporter spec with placement smithi088=a;smithi138=b;count:2 2024-03-26T21:21:41.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:41.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:41 smithi088 ceph-mon[39202]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 126 op/s 2024-03-26T21:21:41.901 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:43.034 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.034 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.034 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:43.035 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:43.035 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:42 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.035 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:42 smithi138 ceph-mon[33706]: Deploying daemon prometheus.a on smithi138 2024-03-26T21:21:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[30075]: Deploying daemon prometheus.a on smithi138 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:21:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:21:43.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:43.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:42 smithi088 ceph-mon[39202]: Deploying daemon prometheus.a on smithi138 2024-03-26T21:21:43.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.599+0000 7f46d5624700 1 -- 172.21.15.138:0/810446226 >> v1:172.21.15.138:6789/0 conn(0x7f46d01094d0 legacy=0x7f46d010b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:43.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.599+0000 7f46d5624700 1 -- 172.21.15.138:0/810446226 shutdown_connections 2024-03-26T21:21:43.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.599+0000 7f46d5624700 1 -- 172.21.15.138:0/810446226 >> 172.21.15.138:0/810446226 conn(0x7f46d00fc900 msgr2=0x7f46d00fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:43.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.599+0000 7f46d5624700 1 -- 172.21.15.138:0/810446226 shutdown_connections 2024-03-26T21:21:43.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.599+0000 7f46d5624700 1 -- 172.21.15.138:0/810446226 wait complete. 2024-03-26T21:21:43.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.600+0000 7f46d5624700 1 Processor -- start 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.600+0000 7f46d5624700 1 -- start start 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.600+0000 7f46d5624700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f46d0198600 con 0x7f46d01094d0 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.600+0000 7f46d5624700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f46d01a3df0 con 0x7f46d0101360 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.600+0000 7f46d5624700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f46d01a4fd0 con 0x7f46d0105790 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.600+0000 7f46ceffd700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f46d0101360 0x7f46d0197ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47292/0 (socket says 172.21.15.138:47292) 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.600+0000 7f46ceffd700 1 -- 172.21.15.138:0/2544439091 learned_addr learned my addr 172.21.15.138:0/2544439091 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194435615 0 0) 0x7f46d01a3df0 con 0x7f46d0101360 2024-03-26T21:21:43.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b80035b0 con 0x7f46d0101360 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896459468 0 0) 0x7f46d01a4fd0 con 0x7f46d0105790 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46d01a3df0 con 0x7f46d0105790 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351086971 0 0) 0x7f46d0198600 con 0x7f46d01094d0 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46d01a4fd0 con 0x7f46d01094d0 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2143792866 0 0) 0x7f46b80035b0 con 0x7f46d0101360 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46d0198600 con 0x7f46d0101360 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f46c0001940 con 0x7f46d0101360 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1505993423 0 0) 0x7f46d01a3df0 con 0x7f46d0105790 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46b80035b0 con 0x7f46d0105790 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f46c40017e0 con 0x7f46d0105790 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2896124480 0 0) 0x7f46d01a4fd0 con 0x7f46d01094d0 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46d01a3df0 con 0x7f46d01094d0 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f46c80017b0 con 0x7f46d01094d0 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3639967736 0 0) 0x7f46d0198600 con 0x7f46d0101360 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 >> v1:172.21.15.88:6790/0 conn(0x7f46d0105790 legacy=0x7f46d019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 >> v1:172.21.15.88:6789/0 conn(0x7f46d01094d0 legacy=0x7f46d01a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:43.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d01a61f0 con 0x7f46d0101360 2024-03-26T21:21:43.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46d01a5200 con 0x7f46d0101360 2024-03-26T21:21:43.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f46d01a5740 con 0x7f46d0101360 2024-03-26T21:21:43.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f46c0001d10 con 0x7f46d0101360 2024-03-26T21:21:43.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.601+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f46c0005e10 con 0x7f46d0101360 2024-03-26T21:21:43.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.602+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46bc005150 con 0x7f46d0101360 2024-03-26T21:21:43.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.603+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 16) ==== 92490+0+0 (unknown 196927556 0 0) 0x7f46c001ca40 con 0x7f46d0101360 2024-03-26T21:21:43.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.603+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(68..68 src has 1..68) ==== 5942+0+0 (unknown 1034119651 0 0) 0x7f46c0089cf0 con 0x7f46d0101360 2024-03-26T21:21:43.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.610+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f46c0057120 con 0x7f46d0101360 2024-03-26T21:21:43.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.819+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi088=a", "target": ["mon-mgr", ""]}) -- 0x7f46bc000bc0 con 0x7f46b806f590 2024-03-26T21:21:43.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.825+0000 7f46b7fff700 1 -- 172.21.15.138:0/2544439091 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (unknown 0 0 1850065467) 0x7f46bc000bc0 con 0x7f46b806f590 2024-03-26T21:21:43.819 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled alertmanager update... 2024-03-26T21:21:43.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.828+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 >> v1:172.21.15.88:6800/4097082509 conn(0x7f46b806f590 legacy=0x7f46b8071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:43.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.828+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 >> v1:172.21.15.138:6789/0 conn(0x7f46d0101360 legacy=0x7f46d0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:43.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.828+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 shutdown_connections 2024-03-26T21:21:43.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.828+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 >> 172.21.15.138:0/2544439091 conn(0x7f46d00fc900 msgr2=0x7f46d006b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:43.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.828+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 shutdown_connections 2024-03-26T21:21:43.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:43.828+0000 7f46d5624700 1 -- 172.21.15.138:0/2544439091 wait complete. 2024-03-26T21:21:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:43 smithi138 ceph-mon[33706]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-26T21:21:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:43 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:44.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:43 smithi088 ceph-mon[30075]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-26T21:21:44.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:43 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:43 smithi088 ceph-mon[39202]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-26T21:21:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:43 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:44.727 DEBUG:teuthology.orchestra.run.smithi088:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T21:21:44.730 INFO:tasks.cephadm:Adding grafana.a on smithi138 2024-03-26T21:21:44.730 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi138=a' 2024-03-26T21:21:44.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:44 smithi138 ceph-mon[33706]: from='client.24481 v1:172.21.15.138:0/2544439091' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi088=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:44.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:44 smithi138 ceph-mon[33706]: Saving service alertmanager spec with placement smithi088=a;count:1 2024-03-26T21:21:44.976 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:44 smithi088 ceph-mon[39202]: from='client.24481 v1:172.21.15.138:0/2544439091' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi088=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:44 smithi088 ceph-mon[39202]: Saving service alertmanager spec with placement smithi088=a;count:1 2024-03-26T21:21:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:44 smithi088 ceph-mon[30075]: from='client.24481 v1:172.21.15.138:0/2544439091' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi088=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:44 smithi088 ceph-mon[30075]: Saving service alertmanager spec with placement smithi088=a;count:1 2024-03-26T21:21:46.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:45 smithi138 ceph-mon[33706]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.6 KiB/s wr, 79 op/s 2024-03-26T21:21:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:45 smithi088 ceph-mon[39202]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.6 KiB/s wr, 79 op/s 2024-03-26T21:21:46.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:45 smithi088 ceph-mon[30075]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.6 KiB/s wr, 79 op/s 2024-03-26T21:21:47.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:21:46 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:21:47.154 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:47 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-26T21:21:47.154 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:47 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T21:21:47.154 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:47 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-26T21:21:47.154 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:47 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-26T21:21:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-26T21:21:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T21:21:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-26T21:21:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-26T21:21:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-26T21:21:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T21:21:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-26T21:21:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:47 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.496+0000 7f3680115700 1 -- 172.21.15.138:0/874994758 >> v1:172.21.15.138:6789/0 conn(0x7f3678075250 legacy=0x7f367810b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.496+0000 7f3680115700 1 -- 172.21.15.138:0/874994758 shutdown_connections 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.496+0000 7f3680115700 1 -- 172.21.15.138:0/874994758 >> 172.21.15.138:0/874994758 conn(0x7f36780fda10 msgr2=0x7f36780ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.496+0000 7f3680115700 1 -- 172.21.15.138:0/874994758 shutdown_connections 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.496+0000 7f3680115700 1 -- 172.21.15.138:0/874994758 wait complete. 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.496+0000 7f3680115700 1 Processor -- start 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f3680115700 1 -- start start 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f3680115700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f367819a700 con 0x7f36780765f0 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f3680115700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f36781a5ef0 con 0x7f3678075250 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f3680115700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f36781a70d0 con 0x7f367810ca10 2024-03-26T21:21:47.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f367deb1700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f3678075250 0x7f3678199bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47306/0 (socket says 172.21.15.138:47306) 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f367deb1700 1 -- 172.21.15.138:0/3658185353 learned_addr learned my addr 172.21.15.138:0/3658185353 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162733371 0 0) 0x7f36781a5ef0 con 0x7f3678075250 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36640035b0 con 0x7f3678075250 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075235083 0 0) 0x7f367819a700 con 0x7f36780765f0 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36781a5ef0 con 0x7f36780765f0 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152356949 0 0) 0x7f36781a70d0 con 0x7f367810ca10 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f367819a700 con 0x7f367810ca10 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4257828624 0 0) 0x7f36640035b0 con 0x7f3678075250 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f36781a70d0 con 0x7f3678075250 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f36740018b0 con 0x7f3678075250 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3061779176 0 0) 0x7f36781a70d0 con 0x7f3678075250 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 >> v1:172.21.15.88:6790/0 conn(0x7f367810ca10 legacy=0x7f36781a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 >> v1:172.21.15.88:6789/0 conn(0x7f36780765f0 legacy=0x7f36781a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:47.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36781a82f0 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f36781a7300 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.497+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f36781a7840 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.498+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f3674003b30 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.498+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3674005ca0 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.498+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f365c005150 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.499+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 16) ==== 92490+0+0 (unknown 196927556 0 0) 0x7f36740036a0 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.499+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(69..69 src has 1..69) ==== 6030+0+0 (unknown 1018206671 0 0) 0x7f3674089ee0 con 0x7f3678075250 2024-03-26T21:21:47.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.504+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f36740571e0 con 0x7f3678075250 2024-03-26T21:21:47.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.732+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 --> v1:172.21.15.88:6800/4097082509 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}) -- 0x7f365c000bc0 con 0x7f366406f620 2024-03-26T21:21:47.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.739+0000 7f366effd700 1 -- 172.21.15.138:0/3658185353 <== mgr.14150 v1:172.21.15.88:6800/4097082509 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (unknown 0 0 664801700) 0x7f365c000bc0 con 0x7f366406f620 2024-03-26T21:21:47.733 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled grafana update... 2024-03-26T21:21:47.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.741+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 >> v1:172.21.15.88:6800/4097082509 conn(0x7f366406f620 legacy=0x7f3664071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:47.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.741+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 >> v1:172.21.15.138:6789/0 conn(0x7f3678075250 legacy=0x7f3678199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:47.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.741+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 shutdown_connections 2024-03-26T21:21:47.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.741+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 >> 172.21.15.138:0/3658185353 conn(0x7f36780fda10 msgr2=0x7f36780fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:47.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.742+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 shutdown_connections 2024-03-26T21:21:47.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:47.742+0000 7f3680115700 1 -- 172.21.15.138:0/3658185353 wait complete. 2024-03-26T21:21:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:21:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.8 KiB/s wr, 49 op/s 2024-03-26T21:21:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-26T21:21:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T21:21:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-26T21:21:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-26T21:21:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T21:21:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:48 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:21:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.8 KiB/s wr, 49 op/s 2024-03-26T21:21:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.8 KiB/s wr, 49 op/s 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-26T21:21:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T21:21:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:48 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:21:48.720 DEBUG:teuthology.orchestra.run.smithi138:grafana.a> sudo journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T21:21:48.722 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T21:21:48.723 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T21:21:48.972 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:21:49.388 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:49 smithi088 ceph-mon[39202]: from='client.24487 v1:172.21.15.138:0/3658185353' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:49.388 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:49 smithi088 ceph-mon[39202]: Saving service grafana spec with placement smithi138=a;count:1 2024-03-26T21:21:49.388 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:49 smithi088 ceph-mon[39202]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T21:21:49.388 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:49 smithi088 ceph-mon[30075]: from='client.24487 v1:172.21.15.138:0/3658185353' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:49.388 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:49 smithi088 ceph-mon[30075]: Saving service grafana spec with placement smithi138=a;count:1 2024-03-26T21:21:49.389 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:49 smithi088 ceph-mon[30075]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T21:21:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:49 smithi138 ceph-mon[33706]: from='client.24487 v1:172.21.15.138:0/3658185353' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:21:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:49 smithi138 ceph-mon[33706]: Saving service grafana spec with placement smithi138=a;count:1 2024-03-26T21:21:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:49 smithi138 ceph-mon[33706]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T21:21:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:50 smithi138 ceph-mon[33706]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:21:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:50 smithi088 ceph-mon[30075]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:21:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:50 smithi088 ceph-mon[39202]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:21:50.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.750+0000 7f9b3831b700 1 -- 172.21.15.88:0/1252286573 >> v1:172.21.15.88:6789/0 conn(0x7f9b30101310 legacy=0x7f9b30101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.750+0000 7f9b3831b700 1 -- 172.21.15.88:0/1252286573 shutdown_connections 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.750+0000 7f9b3831b700 1 -- 172.21.15.88:0/1252286573 >> 172.21.15.88:0/1252286573 conn(0x7f9b300fc8d0 msgr2=0x7f9b300fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.750+0000 7f9b3831b700 1 -- 172.21.15.88:0/1252286573 shutdown_connections 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.750+0000 7f9b3831b700 1 -- 172.21.15.88:0/1252286573 wait complete. 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b3831b700 1 Processor -- start 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b3831b700 1 -- start start 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b3831b700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9b30197240 con 0x7f9b30109480 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b3831b700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9b301a2a30 con 0x7f9b30101310 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b3831b700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9b301a3c10 con 0x7f9b30105740 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b358b6700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9b30105740 0x7f9b3019cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:44554/0 (socket says 172.21.15.88:44554) 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b358b6700 1 -- 172.21.15.88:0/2015514475 learned_addr learned my addr 172.21.15.88:0/2015514475 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2601132692 0 0) 0x7f9b30197240 con 0x7f9b30109480 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b1c005240 con 0x7f9b30109480 2024-03-26T21:21:50.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2475835436 0 0) 0x7f9b301a3c10 con 0x7f9b30105740 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.751+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b30197240 con 0x7f9b30105740 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 69543359 0 0) 0x7f9b301a2a30 con 0x7f9b30101310 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b301a3c10 con 0x7f9b30101310 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2482563193 0 0) 0x7f9b1c005240 con 0x7f9b30109480 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b301a2a30 con 0x7f9b30109480 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9b28001590 con 0x7f9b30109480 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 110123737 0 0) 0x7f9b30197240 con 0x7f9b30105740 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b1c005240 con 0x7f9b30105740 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9b20001870 con 0x7f9b30105740 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2589831326 0 0) 0x7f9b301a3c10 con 0x7f9b30101310 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b30197240 con 0x7f9b30101310 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9b2c001860 con 0x7f9b30101310 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2424472165 0 0) 0x7f9b301a2a30 con 0x7f9b30109480 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 >> v1:172.21.15.88:6790/0 conn(0x7f9b30105740 legacy=0x7f9b3019cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 >> v1:172.21.15.138:6789/0 conn(0x7f9b30101310 legacy=0x7f9b301966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b301a4e30 con 0x7f9b30109480 2024-03-26T21:21:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9b301a3e40 con 0x7f9b30109480 2024-03-26T21:21:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9b301a4380 con 0x7f9b30109480 2024-03-26T21:21:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f9b28003e90 con 0x7f9b30109480 2024-03-26T21:21:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.752+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9b28004d70 con 0x7f9b30109480 2024-03-26T21:21:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.753+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b14005150 con 0x7f9b30109480 2024-03-26T21:21:50.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.754+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 16) ==== 92490+0+0 (unknown 196927556 0 0) 0x7f9b28003a00 con 0x7f9b30109480 2024-03-26T21:21:50.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.754+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 6030+0+0 (unknown 1143573751 0 0) 0x7f9b28088cc0 con 0x7f9b30109480 2024-03-26T21:21:50.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:50.759+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f9b280b9c30 con 0x7f9b30109480 2024-03-26T21:21:51.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.039+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9b14005440 con 0x7f9b30109480 2024-03-26T21:21:51.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.041+0000 7f9b277fe700 1 -- 172.21.15.88:0/2015514475 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (unknown 57899324 0 3507228626) 0x7f9b28056080 con 0x7f9b30109480 2024-03-26T21:21:51.034 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-03-26T21:21:51.034 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQBvPANmGHltAhAAHJXrOSGcswIF1CLvF1X42Q== 2024-03-26T21:21:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.043+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 >> v1:172.21.15.88:6800/4097082509 conn(0x7f9b1c06f540 legacy=0x7f9b1c071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.043+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 >> v1:172.21.15.88:6789/0 conn(0x7f9b30109480 legacy=0x7f9b301a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.043+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 shutdown_connections 2024-03-26T21:21:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.043+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 >> 172.21.15.88:0/2015514475 conn(0x7f9b300fc8d0 msgr2=0x7f9b30108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.043+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 shutdown_connections 2024-03-26T21:21:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:51.043+0000 7f9b3831b700 1 -- 172.21.15.88:0/2015514475 wait complete. 2024-03-26T21:21:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2015514475' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2015514475' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:21:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2015514475' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2015514475' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:21:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2015514475' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2015514475' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:21:51.791 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:21:51.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T21:21:51.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T21:21:51.833 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T21:21:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:21:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:52 smithi138 ceph-mon[33706]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:21:52.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:52 smithi088 ceph-mon[30075]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:21:52.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:52 smithi088 ceph-mon[39202]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.032+0000 7fd2ceb08700 1 -- 172.21.15.138:0/1764016725 >> v1:172.21.15.138:6789/0 conn(0x7fd2c8102ee0 legacy=0x7fd2c81052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.032+0000 7fd2ceb08700 1 -- 172.21.15.138:0/1764016725 shutdown_connections 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.032+0000 7fd2ceb08700 1 -- 172.21.15.138:0/1764016725 >> 172.21.15.138:0/1764016725 conn(0x7fd2c80fc910 msgr2=0x7fd2c80fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.032+0000 7fd2ceb08700 1 -- 172.21.15.138:0/1764016725 shutdown_connections 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.032+0000 7fd2ceb08700 1 -- 172.21.15.138:0/1764016725 wait complete. 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2ceb08700 1 Processor -- start 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2ceb08700 1 -- start start 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2ceb08700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2c81981a0 con 0x7fd2c810a590 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2ceb08700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2c81a3940 con 0x7fd2c8102ee0 2024-03-26T21:21:54.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2ceb08700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2c81a4b20 con 0x7fd2c81069f0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2cc8a4700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fd2c8102ee0 0x7fd2c8197650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:58864/0 (socket says 172.21.15.138:58864) 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2cc8a4700 1 -- 172.21.15.138:0/3149655464 learned_addr learned my addr 172.21.15.138:0/3149655464 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3720227823 0 0) 0x7fd2c81a3940 con 0x7fd2c8102ee0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b00035b0 con 0x7fd2c8102ee0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161859307 0 0) 0x7fd2c81a4b20 con 0x7fd2c81069f0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2c81a3940 con 0x7fd2c81069f0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730708943 0 0) 0x7fd2c81981a0 con 0x7fd2c810a590 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2c81a4b20 con 0x7fd2c810a590 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.033+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2843607757 0 0) 0x7fd2b00035b0 con 0x7fd2c8102ee0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2c81981a0 con 0x7fd2c8102ee0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd2b8004740 con 0x7fd2c8102ee0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2353780063 0 0) 0x7fd2c81981a0 con 0x7fd2c8102ee0 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 >> v1:172.21.15.88:6790/0 conn(0x7fd2c81069f0 legacy=0x7fd2c819dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 >> v1:172.21.15.88:6789/0 conn(0x7fd2c810a590 legacy=0x7fd2c81a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:54.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c81a5d40 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c81a4d50 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fd2b8001bf0 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd2b8005340 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.034+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2c81a5210 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.035+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c810fe80 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.035+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 16) ==== 92490+0+0 (unknown 196927556 0 0) 0x7fd2b8005590 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.036+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 6030+0+0 (unknown 1143573751 0 0) 0x7fd2b808a170 con 0x7fd2c8102ee0 2024-03-26T21:21:54.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.040+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fd2b8057550 con 0x7fd2c8102ee0 2024-03-26T21:21:54.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.342+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fd2c810f6c0 con 0x7fd2c8102ee0 2024-03-26T21:21:54.337 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:54 smithi138 ceph-mon[33706]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T21:21:54.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.347+0000 7fd2c5ffb700 1 -- 172.21.15.138:0/3149655464 <== mon.1 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) ==== 170+0+59 (unknown 3508017507 0 54159229) 0x7fd2c810f6c0 con 0x7fd2c8102ee0 2024-03-26T21:21:54.341 INFO:teuthology.orchestra.run.smithi138.stdout:[client.1] 2024-03-26T21:21:54.341 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQByPANmU9SHFBAAcqR8ObodR8AeITClBBOKig== 2024-03-26T21:21:54.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.349+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 >> v1:172.21.15.88:6800/4097082509 conn(0x7fd2b006f000 legacy=0x7fd2b00714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:54.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.349+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 >> v1:172.21.15.138:6789/0 conn(0x7fd2c8102ee0 legacy=0x7fd2c8197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:54.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.349+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 shutdown_connections 2024-03-26T21:21:54.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.349+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 >> 172.21.15.138:0/3149655464 conn(0x7fd2c80fc910 msgr2=0x7fd2c80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:54.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.349+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 shutdown_connections 2024-03-26T21:21:54.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-26T21:21:54.350+0000 7fd2ceb08700 1 -- 172.21.15.138:0/3149655464 wait complete. 2024-03-26T21:21:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:54 smithi088 ceph-mon[39202]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T21:21:54.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:54 smithi088 ceph-mon[30075]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T21:21:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.138:0/3149655464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:55 smithi138 ceph-mon[33706]: from='client.24499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:55 smithi138 ceph-mon[33706]: from='client.24499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:21:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.138:0/3149655464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:55 smithi088 ceph-mon[39202]: from='client.24499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:55 smithi088 ceph-mon[39202]: from='client.24499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:21:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.138:0/3149655464' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:55 smithi088 ceph-mon[30075]: from='client.24499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:21:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:55 smithi088 ceph-mon[30075]: from='client.24499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:21:55.979 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-26T21:21:55.979 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T21:21:55.980 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T21:21:56.022 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T21:21:56.023 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T21:21:56.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T21:21:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:56 smithi138 ceph-mon[33706]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T21:21:56.273 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:21:56.395 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:56 smithi088 ceph-mon[39202]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T21:21:56.396 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:56 smithi088 ceph-mon[30075]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T21:21:57.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:21:56 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:21:58.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.025+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/3811343359 >> v1:172.21.15.88:6790/0 conn(0x7fb1d0101530 legacy=0x7fb1d0101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:58.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.025+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/3811343359 shutdown_connections 2024-03-26T21:21:58.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.025+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/3811343359 >> 172.21.15.88:0/3811343359 conn(0x7fb1d00fcde0 msgr2=0x7fb1d00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:58.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.025+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/3811343359 shutdown_connections 2024-03-26T21:21:58.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.025+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/3811343359 wait complete. 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.025+0000 7fb1d5a0d700 1 Processor -- start 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.025+0000 7fb1d5a0d700 1 -- start start 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d5a0d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb1d01a74f0 con 0x7fb1d010a1c0 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d5a0d700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb1d01a8710 con 0x7fb1d0101530 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d5a0d700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb1d01a9950 con 0x7fb1d010dd60 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1cf7fe700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fb1d010dd60 0x7fb1d01a5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:48490/0 (socket says 172.21.15.88:48490) 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1cf7fe700 1 -- 172.21.15.88:0/737033348 learned_addr learned my addr 172.21.15.88:0/737033348 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 288916141 0 0) 0x7fb1d01a9950 con 0x7fb1d010dd60 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1bc0035b0 con 0x7fb1d010dd60 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109123480 0 0) 0x7fb1d01a8710 con 0x7fb1d0101530 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d01a9950 con 0x7fb1d0101530 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3328623751 0 0) 0x7fb1d01a74f0 con 0x7fb1d010a1c0 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d01a8710 con 0x7fb1d010a1c0 2024-03-26T21:21:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4073914592 0 0) 0x7fb1bc0035b0 con 0x7fb1d010dd60 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1d01a74f0 con 0x7fb1d010dd60 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.026+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb1c8001d10 con 0x7fb1d010dd60 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1173759695 0 0) 0x7fb1d01a9950 con 0x7fb1d0101530 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1bc0035b0 con 0x7fb1d0101530 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb1c0001910 con 0x7fb1d0101530 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3978291272 0 0) 0x7fb1d01a8710 con 0x7fb1d010a1c0 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1d01a9950 con 0x7fb1d010a1c0 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb1b8001870 con 0x7fb1d010a1c0 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3349944748 0 0) 0x7fb1d01a74f0 con 0x7fb1d010dd60 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 >> v1:172.21.15.138:6789/0 conn(0x7fb1d0101530 legacy=0x7fb1d006a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 >> v1:172.21.15.88:6789/0 conn(0x7fb1d010a1c0 legacy=0x7fb1d01a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d01aab90 con 0x7fb1d010dd60 2024-03-26T21:21:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d01a7720 con 0x7fb1d010dd60 2024-03-26T21:21:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1d01a7be0 con 0x7fb1d010dd60 2024-03-26T21:21:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fb1c8002000 con 0x7fb1d010dd60 2024-03-26T21:21:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.027+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb1c8005fa0 con 0x7fb1d010dd60 2024-03-26T21:21:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.028+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b0005120 con 0x7fb1d010dd60 2024-03-26T21:21:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.028+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 16) ==== 92490+0+0 (unknown 196927556 0 0) 0x7fb1c801c900 con 0x7fb1d010dd60 2024-03-26T21:21:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.029+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(70..70 src has 1..70) ==== 6030+0+0 (unknown 1143573751 0 0) 0x7fb1c8089ee0 con 0x7fb1d010dd60 2024-03-26T21:21:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.034+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fb1c80572a0 con 0x7fb1d010dd60 2024-03-26T21:21:58.183 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:58 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:21:58.183 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:58 smithi088 ceph-mon[30075]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-03-26T21:21:58.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.295+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb1b0005410 con 0x7fb1d010dd60 2024-03-26T21:21:58.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.298+0000 7fb1d4a0b700 1 -- 172.21.15.88:0/737033348 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) ==== 74+0+176737 (unknown 3479712753 0 3041569887) 0x7fb1c8056ba0 con 0x7fb1d010dd60 2024-03-26T21:21:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:21:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.301+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 >> v1:172.21.15.88:6800/4097082509 conn(0x7fb1bc06f5e0 legacy=0x7fb1bc071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.301+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 >> v1:172.21.15.88:6790/0 conn(0x7fb1d010dd60 legacy=0x7fb1d01a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:21:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.302+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 shutdown_connections 2024-03-26T21:21:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.302+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 >> 172.21.15.88:0/737033348 conn(0x7fb1d00fcde0 msgr2=0x7fb1d0111260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:21:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.302+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 shutdown_connections 2024-03-26T21:21:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:21:58.302+0000 7fb1d5a0d700 1 -- 172.21.15.88:0/737033348 wait complete. 2024-03-26T21:21:58.356 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:58 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:21:58.356 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:58 smithi138 ceph-mon[33706]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-03-26T21:21:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:58 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:21:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:58 smithi088 ceph-mon[39202]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-03-26T21:21:58.914 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":4097082509}]},"active_addr":"172.21.15.88:6800/4097082509","active_change":"2024-03-26T21:16:47.065176+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.88:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":3138370048}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":234365906}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":670122773}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":4136937701}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":853869010}]}]} 2024-03-26T21:21:58.919 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T21:21:58.919 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T21:21:58.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:21:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:21:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:21:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/737033348' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T21:21:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:21:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/737033348' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T21:21:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:21:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/737033348' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T21:22:00.430 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:00 smithi088 ceph-mon[39202]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-26T21:22:00.430 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:00 smithi088 ceph-mon[30075]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-26T21:22:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-mon[33706]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-26T21:22:00.474 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 systemd[1]: Starting Ceph prometheus.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:22:00.843 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 podman[64509]: 2024-03-26 21:22:00.59929622 +0000 UTC m=+0.104201053 container create b166b60e3e9ee4384cf835982f181a515be5d660fb19643b03f54c92541bceab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:22:00.843 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 podman[64509]: 2024-03-26 21:22:00.513812449 +0000 UTC m=+0.018717289 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T21:22:00.843 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 podman[64509]: 2024-03-26 21:22:00.818287001 +0000 UTC m=+0.323191835 container init b166b60e3e9ee4384cf835982f181a515be5d660fb19643b03f54c92541bceab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:22:00.843 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 podman[64509]: 2024-03-26 21:22:00.823955221 +0000 UTC m=+0.328860056 container start b166b60e3e9ee4384cf835982f181a515be5d660fb19643b03f54c92541bceab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:22:00.843 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.848Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.848Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.848Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi138 (none))" 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.848Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.848Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.849Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.849Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.850Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T21:22:00.844 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.850Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T21:22:00.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.942+0000 7ff92c2c4700 1 -- 172.21.15.88:0/1123904541 >> v1:172.21.15.88:6790/0 conn(0x7ff924101530 legacy=0x7ff924101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.942+0000 7ff92c2c4700 1 -- 172.21.15.88:0/1123904541 shutdown_connections 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.942+0000 7ff92c2c4700 1 -- 172.21.15.88:0/1123904541 >> 172.21.15.88:0/1123904541 conn(0x7ff9240fcde0 msgr2=0x7ff9240ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.942+0000 7ff92c2c4700 1 -- 172.21.15.88:0/1123904541 shutdown_connections 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.942+0000 7ff92c2c4700 1 -- 172.21.15.88:0/1123904541 wait complete. 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff92c2c4700 1 Processor -- start 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff92c2c4700 1 -- start start 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff92c2c4700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9241a7530 con 0x7ff92410dd60 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff92c2c4700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9241a8750 con 0x7ff92410a1c0 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff92c2c4700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9241a9990 con 0x7ff924101530 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff92a060700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff924101530 0x7ff92406a9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:48510/0 (socket says 172.21.15.88:48510) 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff92a060700 1 -- 172.21.15.88:0/2283120030 learned_addr learned my addr 172.21.15.88:0/2283120030 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:00.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594514782 0 0) 0x7ff9241a7530 con 0x7ff92410dd60 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff90c0035b0 con 0x7ff92410dd60 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675667568 0 0) 0x7ff9241a9990 con 0x7ff924101530 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9241a7530 con 0x7ff924101530 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645360032 0 0) 0x7ff9241a8750 con 0x7ff92410a1c0 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9241a9990 con 0x7ff92410a1c0 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2702121422 0 0) 0x7ff90c0035b0 con 0x7ff92410dd60 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9241a8750 con 0x7ff92410dd60 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.943+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff9140018b0 con 0x7ff92410dd60 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3694908858 0 0) 0x7ff9241a7530 con 0x7ff924101530 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff90c0035b0 con 0x7ff924101530 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff91c0048c0 con 0x7ff924101530 2024-03-26T21:22:00.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2472205398 0 0) 0x7ff9241a9990 con 0x7ff92410a1c0 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9241a7530 con 0x7ff92410a1c0 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff918001110 con 0x7ff92410a1c0 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1640344499 0 0) 0x7ff9241a8750 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 >> v1:172.21.15.88:6790/0 conn(0x7ff924101530 legacy=0x7ff92406a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 >> v1:172.21.15.138:6789/0 conn(0x7ff92410a1c0 legacy=0x7ff9241a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9241aabd0 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9241a7760 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9241a7c20 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7ff914003b30 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.944+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7ff914004b00 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.945+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff910005120 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.945+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 16) ==== 92490+0+0 (unknown 196927556 0 0) 0x7ff9140036a0 con 0x7ff92410dd60 2024-03-26T21:22:00.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.946+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 6030+0+0 (unknown 1143573751 0 0) 0x7ff914088b60 con 0x7ff92410dd60 2024-03-26T21:22:00.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:00.950+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7ff914055f40 con 0x7ff92410dd60 2024-03-26T21:22:01.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.152+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff910005410 con 0x7ff92410dd60 2024-03-26T21:22:01.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.154+0000 7ff9237fe700 1 -- 172.21.15.88:0/2283120030 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+20717 (unknown 2276865964 0 3096196357) 0x7ff914055840 con 0x7ff92410dd60 2024-03-26T21:22:01.147 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:22:01.148 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":70,"fsid":"e240a1a4-ebb5-11ee-95d0-87774f69a715","created":"2024-03-26T21:15:46.786114+0000","modified":"2024-03-26T21:21:48.160125+0000","last_up_change":"2024-03-26T21:21:11.241387+0000","last_in_change":"2024-03-26T21:20:53.506385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:19:03.150113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T21:21:24.098642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T21:21:24.132686+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T21:21:25.253554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T21:21:27.214161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T21:21:29.267071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09199105-49c9-4d89-8c77-9193d4a78bf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":3845975585}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":3845975585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":3845975585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":3845975585}]},"public_addr":"172.21.15.88:6801/3845975585","cluster_addr":"172.21.15.88:6802/3845975585","heartbeat_back_addr":"172.21.15.88:6804/3845975585","heartbeat_front_addr":"172.21.15.88:6803/3845975585","state":["exists","up"]},{"osd":1,"uuid":"e2759e98-1df5-4e4d-a7f0-719202f0bc5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":698625447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":698625447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":698625447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":698625447}]},"public_addr":"172.21.15.88:6805/698625447","cluster_addr":"172.21.15.88:6806/698625447","heartbeat_back_addr":"172.21.15.88:6808/698625447","heartbeat_front_addr":"172.21.15.88:6807/698625447","state":["exists","up"]},{"osd":2,"uuid":"5bb8665c-3450-49dc-86fb-0d086ffb8ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":2517903952}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":2517903952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":2517903952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":2517903952}]},"public_addr":"172.21.15.88:6809/2517903952","cluster_addr":"172.21.15.88:6810/2517903952","heartbeat_back_addr":"172.21.15.88:6812/2517903952","heartbeat_front_addr":"172.21.15.88:6811/2517903952","state":["exists","up"]},{"osd":3,"uuid":"a727a6dd-71c5-4867-a2ad-91423368c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":3938069994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":3938069994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6816","nonce":3938069994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":3938069994}]},"public_addr":"172.21.15.88:6813/3938069994","cluster_addr":"172.21.15.88:6814/3938069994","heartbeat_back_addr":"172.21.15.88:6816/3938069994","heartbeat_front_addr":"172.21.15.88:6815/3938069994","state":["exists","up"]},{"osd":4,"uuid":"a2a562d8-4071-44c7-9695-7652271cbd52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6800","nonce":2187218446}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6801","nonce":2187218446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6803","nonce":2187218446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6802","nonce":2187218446}]},"public_addr":"172.21.15.138:6800/2187218446","cluster_addr":"172.21.15.138:6801/2187218446","heartbeat_back_addr":"172.21.15.138:6803/2187218446","heartbeat_front_addr":"172.21.15.138:6802/2187218446","state":["exists","up"]},{"osd":5,"uuid":"6c2c5207-7f0a-4700-b8e1-afeba8499378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6804","nonce":2329820906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6805","nonce":2329820906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6807","nonce":2329820906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6806","nonce":2329820906}]},"public_addr":"172.21.15.138:6804/2329820906","cluster_addr":"172.21.15.138:6805/2329820906","heartbeat_back_addr":"172.21.15.138:6807/2329820906","heartbeat_front_addr":"172.21.15.138:6806/2329820906","state":["exists","up"]},{"osd":6,"uuid":"2dbb6451-7d30-4039-9b3d-f01e9acd00c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6808","nonce":771295543}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6809","nonce":771295543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6811","nonce":771295543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6810","nonce":771295543}]},"public_addr":"172.21.15.138:6808/771295543","cluster_addr":"172.21.15.138:6809/771295543","heartbeat_back_addr":"172.21.15.138:6811/771295543","heartbeat_front_addr":"172.21.15.138:6810/771295543","state":["exists","up"]},{"osd":7,"uuid":"e0ba8a7d-0c56-4b23-838a-866cad1b91a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6812","nonce":1004436505}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6813","nonce":1004436505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6815","nonce":1004436505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6814","nonce":1004436505}]},"public_addr":"172.21.15.138:6812/1004436505","cluster_addr":"172.21.15.138:6813/1004436505","heartbeat_back_addr":"172.21.15.138:6815/1004436505","heartbeat_front_addr":"172.21.15.138:6814/1004436505","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:18:09.042877+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:18:34.576623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:00.195601+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:26.199781+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:50.484242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:20:17.097720+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:20:43.230683+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:21:10.028091+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":7,"to":4}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6800/3808082170":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/2491754424":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/467543406":"2024-03-27T21:16:20.042822+0000","172.21.15.88:6800/221932617":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/2048865678":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/2427433423":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/1032838430":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/2073122482":"2024-03-27T21:16:47.065067+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:22:01.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.156+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 >> v1:172.21.15.88:6800/4097082509 conn(0x7ff90c06f570 legacy=0x7ff90c071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:01.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.156+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 >> v1:172.21.15.88:6789/0 conn(0x7ff92410dd60 legacy=0x7ff9241a5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:01.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.156+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 shutdown_connections 2024-03-26T21:22:01.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.156+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 >> 172.21.15.88:0/2283120030 conn(0x7ff9240fcde0 msgr2=0x7ff924111240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:01.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.156+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 shutdown_connections 2024-03-26T21:22:01.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:01.156+0000 7ff92c2c4700 1 -- 172.21.15.88:0/2283120030 wait complete. 2024-03-26T21:22:01.224 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.852Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T21:22:01.224 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.852Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=10.912µs 2024-03-26T21:22:01.224 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.852Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T21:22:01.224 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.853Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T21:22:01.224 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.853Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.266µs wal_replay_duration=273.091µs wbl_replay_duration=142ns total_replay_duration=398.682µs 2024-03-26T21:22:01.224 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.854Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T21:22:01.225 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.854Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T21:22:01.225 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.854Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T21:22:01.225 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.874Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.509727ms db_storage=732ns remote_storage=1.095µs web_handler=330ns query_engine=535ns scrape=389.309µs scrape_sd=85.325µs notify=710ns notify_sd=1.393µs rules=19.77468ms tracing=5.82µs 2024-03-26T21:22:01.225 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.874Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T21:22:01.225 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:22:00.874Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T21:22:01.225 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 bash[64509]: b166b60e3e9ee4384cf835982f181a515be5d660fb19643b03f54c92541bceab 2024-03-26T21:22:01.225 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:22:00 smithi138 systemd[1]: Started Ceph prometheus.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:22:01.967 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T21:22:01.967 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:22:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:02.233 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ignoring --setuser ceph since I am not root 2024-03-26T21:22:02.233 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ignoring --setgroup ceph since I am not root 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[39202]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 0 objects/s recovering 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2283120030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T21:22:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[30075]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 0 objects/s recovering 2024-03-26T21:22:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2283120030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:22:02.474 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: ignoring --setuser ceph since I am not root 2024-03-26T21:22:02.474 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: ignoring --setgroup ceph since I am not root 2024-03-26T21:22:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:01 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:01 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:01 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' 2024-03-26T21:22:02.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:01 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T21:22:02.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:01 smithi138 ceph-mon[33706]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 0 objects/s recovering 2024-03-26T21:22:02.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2283120030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:22:02.616 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:02.623+0000 7fab13f94200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:22:02.974 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:02.600+0000 7ff0c44e5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:22:02.974 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:02.692+0000 7ff0c44e5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:22:02.983 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:02.728+0000 7fab13f94200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:22:03.328 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:03 smithi138 ceph-mon[33706]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T21:22:03.329 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:03 smithi138 ceph-mon[33706]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T21:22:03.371 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-mon[39202]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T21:22:03.371 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-mon[39202]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T21:22:03.371 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-mon[30075]: from='mgr.14150 v1:172.21.15.88:0/2082342482' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T21:22:03.371 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-mon[30075]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T21:22:03.625 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:03.377+0000 7fab13f94200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:22:03.625 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:03.472+0000 7fab13f94200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:22:03.625 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:03.631+0000 7fab13f94200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:22:03.661 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:03 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:03.334+0000 7ff0c44e5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:22:03.662 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:03 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:03.429+0000 7ff0c44e5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:22:03.662 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:03 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:03.571+0000 7ff0c44e5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:22:03.662 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:03 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:03.666+0000 7ff0c44e5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:22:03.880 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:03.729+0000 7fab13f94200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:22:03.880 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:03 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:03.815+0000 7fab13f94200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:22:03.974 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:03 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:03.749+0000 7ff0c44e5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:22:04.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.095+0000 7fe25ecad700 1 -- 172.21.15.88:0/3229328262 >> v1:172.21.15.88:6789/0 conn(0x7fe25810ee80 legacy=0x7fe258111320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:04.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.095+0000 7fe25ecad700 1 -- 172.21.15.88:0/3229328262 shutdown_connections 2024-03-26T21:22:04.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.095+0000 7fe25ecad700 1 -- 172.21.15.88:0/3229328262 >> 172.21.15.88:0/3229328262 conn(0x7fe258077770 msgr2=0x7fe2580feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:04.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.095+0000 7fe25ecad700 1 -- 172.21.15.88:0/3229328262 shutdown_connections 2024-03-26T21:22:04.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.095+0000 7fe25ecad700 1 -- 172.21.15.88:0/3229328262 wait complete. 2024-03-26T21:22:04.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25ecad700 1 Processor -- start 2024-03-26T21:22:04.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25ecad700 1 -- start start 2024-03-26T21:22:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25ecad700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2581079f0 con 0x7fe25810ee80 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25ecad700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe258107c00 con 0x7fe2581084e0 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25ecad700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2581abe70 con 0x7fe25810b250 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25ca49700 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fe2581084e0 0x7fe258101a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.88:50014/0 (socket says 172.21.15.88:50014) 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe257fff700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fe25810b250 0x7fe258102110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47776/0 (socket says 172.21.15.88:47776) 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25d24a700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe25810ee80 0x7fe2581a8730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38500/0 (socket says 172.21.15.88:38500) 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.096+0000 7fe25ca49700 1 -- 172.21.15.88:0/1273027074 learned_addr learned my addr 172.21.15.88:0/1273027074 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920969912 0 0) 0x7fe258107c00 con 0x7fe2581084e0 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2480035b0 con 0x7fe2581084e0 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230689965 0 0) 0x7fe2581079f0 con 0x7fe25810ee80 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe258107c00 con 0x7fe25810ee80 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175953862 0 0) 0x7fe2581abe70 con 0x7fe25810b250 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2581079f0 con 0x7fe25810b250 2024-03-26T21:22:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 534730690 0 0) 0x7fe2480035b0 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2581abe70 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe24c0018b0 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1302978141 0 0) 0x7fe2581079f0 con 0x7fe25810b250 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2480035b0 con 0x7fe25810b250 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3761961631 0 0) 0x7fe258107c00 con 0x7fe25810ee80 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2581079f0 con 0x7fe25810ee80 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe244001830 con 0x7fe25810b250 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe2500017b0 con 0x7fe25810ee80 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2509893336 0 0) 0x7fe2581abe70 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 >> v1:172.21.15.88:6790/0 conn(0x7fe25810b250 legacy=0x7fe258102110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 >> v1:172.21.15.88:6789/0 conn(0x7fe25810ee80 legacy=0x7fe2581a8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2581ad050 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.097+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.138:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2581ac040 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.098+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fe24c003b30 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.098+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe24c005c80 con 0x7fe2581084e0 2024-03-26T21:22:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.098+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.138:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2581ac500 con 0x7fe2581084e0 2024-03-26T21:22:04.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.098+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe258103b40 con 0x7fe2581084e0 2024-03-26T21:22:04.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.099+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 7 ==== mgrmap(e 17) ==== 92504+0+0 (unknown 3627722014 0 0) 0x7fe24c0036a0 con 0x7fe2581084e0 2024-03-26T21:22:04.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.100+0000 7fe257fff700 1 -- 172.21.15.88:0/1273027074 >> v1:172.21.15.88:6800/4097082509 conn(0x7fe24806f740 legacy=0x7fe248071c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/4097082509 2024-03-26T21:22:04.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.100+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 6030+0+0 (unknown 1143573751 0 0) 0x7fe24c089d90 con 0x7fe2581084e0 2024-03-26T21:22:04.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.108+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fe24c057090 con 0x7fe2581084e0 2024-03-26T21:22:04.233 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:04 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:04.091+0000 7fab13f94200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:22:04.233 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:04 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:04.207+0000 7fab13f94200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:22:04.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.301+0000 7fe257fff700 1 -- 172.21.15.88:0/1273027074 >> v1:172.21.15.88:6800/4097082509 conn(0x7fe24806f740 legacy=0x7fe248071c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.88:6800/4097082509 2024-03-26T21:22:04.303 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:04.014+0000 7ff0c44e5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:22:04.303 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:04.128+0000 7ff0c44e5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:22:04.303 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:04.215+0000 7ff0c44e5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:22:04.303 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:04.309+0000 7ff0c44e5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:22:04.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.313+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 --> v1:172.21.15.138:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe258068f30 con 0x7fe2581084e0 2024-03-26T21:22:04.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.315+0000 7fe255ffb700 1 -- 172.21.15.88:0/1273027074 <== mon.1 v1:172.21.15.138:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+20717 (unknown 2276865964 0 3096196357) 0x7fe24c056990 con 0x7fe2581084e0 2024-03-26T21:22:04.309 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:22:04.309 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":70,"fsid":"e240a1a4-ebb5-11ee-95d0-87774f69a715","created":"2024-03-26T21:15:46.786114+0000","modified":"2024-03-26T21:21:48.160125+0000","last_up_change":"2024-03-26T21:21:11.241387+0000","last_in_change":"2024-03-26T21:20:53.506385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:19:03.150113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T21:21:24.098642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T21:21:24.132686+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T21:21:25.253554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T21:21:27.214161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T21:21:29.267071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09199105-49c9-4d89-8c77-9193d4a78bf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":3845975585}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":3845975585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":3845975585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":3845975585}]},"public_addr":"172.21.15.88:6801/3845975585","cluster_addr":"172.21.15.88:6802/3845975585","heartbeat_back_addr":"172.21.15.88:6804/3845975585","heartbeat_front_addr":"172.21.15.88:6803/3845975585","state":["exists","up"]},{"osd":1,"uuid":"e2759e98-1df5-4e4d-a7f0-719202f0bc5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":698625447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":698625447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":698625447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":698625447}]},"public_addr":"172.21.15.88:6805/698625447","cluster_addr":"172.21.15.88:6806/698625447","heartbeat_back_addr":"172.21.15.88:6808/698625447","heartbeat_front_addr":"172.21.15.88:6807/698625447","state":["exists","up"]},{"osd":2,"uuid":"5bb8665c-3450-49dc-86fb-0d086ffb8ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":2517903952}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":2517903952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":2517903952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":2517903952}]},"public_addr":"172.21.15.88:6809/2517903952","cluster_addr":"172.21.15.88:6810/2517903952","heartbeat_back_addr":"172.21.15.88:6812/2517903952","heartbeat_front_addr":"172.21.15.88:6811/2517903952","state":["exists","up"]},{"osd":3,"uuid":"a727a6dd-71c5-4867-a2ad-91423368c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":3938069994}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":3938069994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6816","nonce":3938069994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":3938069994}]},"public_addr":"172.21.15.88:6813/3938069994","cluster_addr":"172.21.15.88:6814/3938069994","heartbeat_back_addr":"172.21.15.88:6816/3938069994","heartbeat_front_addr":"172.21.15.88:6815/3938069994","state":["exists","up"]},{"osd":4,"uuid":"a2a562d8-4071-44c7-9695-7652271cbd52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6800","nonce":2187218446}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6801","nonce":2187218446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6803","nonce":2187218446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6802","nonce":2187218446}]},"public_addr":"172.21.15.138:6800/2187218446","cluster_addr":"172.21.15.138:6801/2187218446","heartbeat_back_addr":"172.21.15.138:6803/2187218446","heartbeat_front_addr":"172.21.15.138:6802/2187218446","state":["exists","up"]},{"osd":5,"uuid":"6c2c5207-7f0a-4700-b8e1-afeba8499378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6804","nonce":2329820906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6805","nonce":2329820906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6807","nonce":2329820906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6806","nonce":2329820906}]},"public_addr":"172.21.15.138:6804/2329820906","cluster_addr":"172.21.15.138:6805/2329820906","heartbeat_back_addr":"172.21.15.138:6807/2329820906","heartbeat_front_addr":"172.21.15.138:6806/2329820906","state":["exists","up"]},{"osd":6,"uuid":"2dbb6451-7d30-4039-9b3d-f01e9acd00c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6808","nonce":771295543}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6809","nonce":771295543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6811","nonce":771295543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6810","nonce":771295543}]},"public_addr":"172.21.15.138:6808/771295543","cluster_addr":"172.21.15.138:6809/771295543","heartbeat_back_addr":"172.21.15.138:6811/771295543","heartbeat_front_addr":"172.21.15.138:6810/771295543","state":["exists","up"]},{"osd":7,"uuid":"e0ba8a7d-0c56-4b23-838a-866cad1b91a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6812","nonce":1004436505}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6813","nonce":1004436505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6815","nonce":1004436505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.138:6814","nonce":1004436505}]},"public_addr":"172.21.15.138:6812/1004436505","cluster_addr":"172.21.15.138:6813/1004436505","heartbeat_back_addr":"172.21.15.138:6815/1004436505","heartbeat_front_addr":"172.21.15.138:6814/1004436505","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:18:09.042877+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:18:34.576623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:00.195601+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:26.199781+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:19:50.484242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:20:17.097720+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:20:43.230683+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:21:10.028091+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":7,"to":4}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6800/3808082170":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/2491754424":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/467543406":"2024-03-27T21:16:20.042822+0000","172.21.15.88:6800/221932617":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/2048865678":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/2427433423":"2024-03-27T21:16:20.042822+0000","172.21.15.88:0/1032838430":"2024-03-27T21:16:47.065067+0000","172.21.15.88:0/2073122482":"2024-03-27T21:16:47.065067+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:22:04.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.318+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 >> v1:172.21.15.88:6800/4097082509 conn(0x7fe24806f740 legacy=0x7fe248071c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T21:22:04.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.318+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 >> v1:172.21.15.138:6789/0 conn(0x7fe2581084e0 legacy=0x7fe258101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:04.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.318+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 shutdown_connections 2024-03-26T21:22:04.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.318+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 >> 172.21.15.88:0/1273027074 conn(0x7fe258077770 msgr2=0x7fe25810fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:04.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.318+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 shutdown_connections 2024-03-26T21:22:04.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:04.318+0000 7fe25ecad700 1 -- 172.21.15.88:0/1273027074 wait complete. 2024-03-26T21:22:04.571 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1273027074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:22:04.571 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1273027074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:22:04.572 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:04 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:04.297+0000 7fab13f94200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:22:04.572 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:04 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:04.398+0000 7fab13f94200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:22:04.724 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:04.491+0000 7ff0c44e5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:22:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1273027074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:22:04.913 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:04 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:04.577+0000 7fab13f94200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:22:04.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T21:22:04.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T21:22:04.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T21:22:04.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T21:22:04.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T21:22:04.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T21:22:04.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T21:22:04.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T21:22:05.531 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:05.531 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:05.531 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:05.540 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:05.540 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:05.541 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:05.544 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:05.724 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:05 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:05.443+0000 7ff0c44e5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:22:05.724 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:05 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:05.572+0000 7ff0c44e5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:22:05.928 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:05 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:05.797+0000 7fab13f94200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:22:05.929 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:05 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:05.935+0000 7fab13f94200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:22:06.183 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:05 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:05.905+0000 7ff0c44e5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:22:06.183 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:05 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:05.994+0000 7ff0c44e5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:22:06.474 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:06 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:06.189+0000 7ff0c44e5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:22:06.474 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:06 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:06.374+0000 7ff0c44e5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:22:06.574 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:06 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:06.280+0000 7fab13f94200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:22:06.574 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:06 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:06.374+0000 7fab13f94200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:22:06.894 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:06 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:06.580+0000 7fab13f94200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:22:06.894 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:06 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:06.769+0000 7fab13f94200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:22:06.918 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:06 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:06.661+0000 7ff0c44e5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:22:07.146 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:07 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:07.068+0000 7fab13f94200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:22:07.224 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:06 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:06.924+0000 7ff0c44e5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:22:07.224 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:07 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:07.163+0000 7ff0c44e5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:22:07.396 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:07 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:07.339+0000 7fab13f94200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:22:07.730 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:07 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:07.599+0000 7fab13f94200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:22:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:08.224 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:07 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:07.970+0000 7ff0c44e5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:22:08.658 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:08.438+0000 7fab13f94200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:22:08.868 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:08.555+0000 7ff0c44e5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:22:08.869 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:08.654+0000 7ff0c44e5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:22:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-mon[33706]: Standby manager daemon x restarted 2024-03-26T21:22:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-mon[33706]: Standby manager daemon x started 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-mon[33706]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:08.875+0000 7ff0c44e5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: [26/Mar/2024:21:22:08] ENGINE Bus STARTING 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: CherryPy Checker: 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: The Application mounted at '' has an empty config. 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: 2024-03-26T21:22:09.225 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: [26/Mar/2024:21:22:08] ENGINE Serving on http://:::9283 2024-03-26T21:22:09.226 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:22:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x[35794]: [26/Mar/2024:21:22:08] ENGINE Bus STARTED 2024-03-26T21:22:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[39202]: Standby manager daemon x restarted 2024-03-26T21:22:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[39202]: Standby manager daemon x started 2024-03-26T21:22:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:22:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:22:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:22:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[39202]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[30075]: Standby manager daemon x restarted 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[30075]: Standby manager daemon x started 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:08 smithi088 ceph-mon[30075]: from='mgr.? v1:172.21.15.138:0/4028257582' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:09.031+0000 7fab13f94200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:22:09.234 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:09.133+0000 7fab13f94200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:22:09.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:09.356+0000 7fab13f94200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:22:09.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:09] ENGINE Bus STARTING 2024-03-26T21:22:09.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: CherryPy Checker: 2024-03-26T21:22:09.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: The Application mounted at '' has an empty config. 2024-03-26T21:22:09.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:22:09.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:09] ENGINE Serving on http://:::9283 2024-03-26T21:22:09.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:09] ENGINE Bus STARTED 2024-03-26T21:22:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T21:22:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: Active manager daemon y restarted 2024-03-26T21:22:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: Activating manager daemon y 2024-03-26T21:22:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: mgrmap e19: y(active, starting, since 0.00905139s), standbys: x 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:22:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: Manager daemon y is now available 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:22:10.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:09 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: Active manager daemon y restarted 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: Activating manager daemon y 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: mgrmap e19: y(active, starting, since 0.00905139s), standbys: x 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:22:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: Manager daemon y is now available 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:22:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: Active manager daemon y restarted 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: Activating manager daemon y 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: mgrmap e19: y(active, starting, since 0.00905139s), standbys: x 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:22:10.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: Manager daemon y is now available 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:22:10.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:10.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:10.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:22:10.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:22:10.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:22:10.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:09 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:22:10.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:22:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:22:11.196 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:10 smithi088 ceph-mon[39202]: [26/Mar/2024:21:22:09] ENGINE Bus STARTING 2024-03-26T21:22:11.196 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:10 smithi088 ceph-mon[39202]: [26/Mar/2024:21:22:10] ENGINE Serving on https://172.21.15.88:7150 2024-03-26T21:22:11.196 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:10 smithi088 ceph-mon[39202]: mgrmap e20: y(active, since 1.01137s), standbys: x 2024-03-26T21:22:11.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:10 smithi088 ceph-mon[30075]: [26/Mar/2024:21:22:09] ENGINE Bus STARTING 2024-03-26T21:22:11.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:10 smithi088 ceph-mon[30075]: [26/Mar/2024:21:22:10] ENGINE Serving on https://172.21.15.88:7150 2024-03-26T21:22:11.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:10 smithi088 ceph-mon[30075]: mgrmap e20: y(active, since 1.01137s), standbys: x 2024-03-26T21:22:11.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:10 smithi138 ceph-mon[33706]: [26/Mar/2024:21:22:09] ENGINE Bus STARTING 2024-03-26T21:22:11.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:10 smithi138 ceph-mon[33706]: [26/Mar/2024:21:22:10] ENGINE Serving on https://172.21.15.88:7150 2024-03-26T21:22:11.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:10 smithi138 ceph-mon[33706]: mgrmap e20: y(active, since 1.01137s), standbys: x 2024-03-26T21:22:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:11 smithi138 ceph-mon[33706]: [26/Mar/2024:21:22:10] ENGINE Serving on http://172.21.15.88:8765 2024-03-26T21:22:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:11 smithi138 ceph-mon[33706]: [26/Mar/2024:21:22:10] ENGINE Bus STARTED 2024-03-26T21:22:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:11 smithi138 ceph-mon[33706]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:12.231 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[39202]: [26/Mar/2024:21:22:10] ENGINE Serving on http://172.21.15.88:8765 2024-03-26T21:22:12.231 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[39202]: [26/Mar/2024:21:22:10] ENGINE Bus STARTED 2024-03-26T21:22:12.231 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:12.231 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[39202]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:12.231 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[30075]: [26/Mar/2024:21:22:10] ENGINE Serving on http://172.21.15.88:8765 2024-03-26T21:22:12.232 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[30075]: [26/Mar/2024:21:22:10] ENGINE Bus STARTED 2024-03-26T21:22:12.232 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:12.232 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:11 smithi088 ceph-mon[30075]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.242+0000 7f692b9d0700 1 -- 172.21.15.88:0/1133841636 >> v1:172.21.15.88:6790/0 conn(0x7f692410ee90 legacy=0x7f6924111330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.242+0000 7f692b9d0700 1 -- 172.21.15.88:0/1133841636 shutdown_connections 2024-03-26T21:22:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.242+0000 7f692b9d0700 1 -- 172.21.15.88:0/1133841636 >> 172.21.15.88:0/1133841636 conn(0x7f69240765d0 msgr2=0x7f69240769e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.242+0000 7f692b9d0700 1 -- 172.21.15.88:0/1133841636 shutdown_connections 2024-03-26T21:22:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.242+0000 7f692b9d0700 1 -- 172.21.15.88:0/1133841636 wait complete. 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.242+0000 7f692b9d0700 1 Processor -- start 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f692b9d0700 1 -- start start 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f692b9d0700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69241a9690 con 0x7f69241084f0 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f692b9d0700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69241aa8b0 con 0x7f692410ee90 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f692b9d0700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69241abad0 con 0x7f692410b260 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f6928f6b700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f692410b260 0x7f69241a4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47874/0 (socket says 172.21.15.88:47874) 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f6928f6b700 1 -- 172.21.15.88:0/235040099 learned_addr learned my addr 172.21.15.88:0/235040099 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f692976c700 1 --1- 172.21.15.88:0/235040099 >> v1:172.21.15.88:6789/0 conn(0x7f69241084f0 0x7f6924107710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38600/0 (socket says 172.21.15.88:38600) 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249734405 0 0) 0x7f69241a9690 con 0x7f69241084f0 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69100035b0 con 0x7f69241084f0 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 238167377 0 0) 0x7f69241abad0 con 0x7f692410b260 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69241a9690 con 0x7f692410b260 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494415211 0 0) 0x7f69241aa8b0 con 0x7f692410ee90 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69241abad0 con 0x7f692410ee90 2024-03-26T21:22:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2219739870 0 0) 0x7f69100035b0 con 0x7f69241084f0 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69241aa8b0 con 0x7f69241084f0 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f69200012d0 con 0x7f69241084f0 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4043665224 0 0) 0x7f69241a9690 con 0x7f692410b260 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69100035b0 con 0x7f692410b260 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.243+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1049367057 0 0) 0x7f69241abad0 con 0x7f692410ee90 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69241a9690 con 0x7f692410ee90 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6914001870 con 0x7f692410b260 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f691c004860 con 0x7f692410ee90 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2359168684 0 0) 0x7f69241aa8b0 con 0x7f69241084f0 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 >> v1:172.21.15.88:6790/0 conn(0x7f692410b260 legacy=0x7f69241a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 >> v1:172.21.15.138:6789/0 conn(0x7f692410ee90 legacy=0x7f69241a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69241acd10 con 0x7f69241084f0 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69241a98c0 con 0x7f69241084f0 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f69241a9d80 con 0x7f69241084f0 2024-03-26T21:22:12.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f6920001580 con 0x7f69241084f0 2024-03-26T21:22:12.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6920005c70 con 0x7f69241084f0 2024-03-26T21:22:12.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.244+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6908000fb0 con 0x7f69241084f0 2024-03-26T21:22:12.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.245+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 20) ==== 92248+0+0 (unknown 135513095 0 0) 0x7f692001c670 con 0x7f69241084f0 2024-03-26T21:22:12.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.246+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f6920089980 con 0x7f69241084f0 2024-03-26T21:22:12.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.246+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6813/3938069994 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f691c001400 con 0x7f6910079c10 2024-03-26T21:22:12.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.246+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f6920089de0 con 0x7f69241084f0 2024-03-26T21:22:12.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.247+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== osd.3 v1:172.21.15.88:6813/3938069994 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f691c001400 con 0x7f6910079c10 2024-03-26T21:22:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.263+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 --> v1:172.21.15.88:6813/3938069994 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6908002d10 con 0x7f6910079c10 2024-03-26T21:22:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f691a7fc700 1 -- 172.21.15.88:0/235040099 <== osd.3 v1:172.21.15.88:6813/3938069994 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3775874274) 0x7f6908002d10 con 0x7f6910079c10 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 >> v1:172.21.15.88:6813/3938069994 conn(0x7f6910079c10 legacy=0x7f691007c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 >> v1:172.21.15.88:6800/3939589528 conn(0x7f691006f430 legacy=0x7f69100718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 >> v1:172.21.15.88:6789/0 conn(0x7f69241084f0 legacy=0x7f6924107710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f6929f6d700 1 -- 172.21.15.88:0/235040099 reap_dead start 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 shutdown_connections 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 >> 172.21.15.88:0/235040099 conn(0x7f69240765d0 msgr2=0x7f6924109230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 shutdown_connections 2024-03-26T21:22:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:12.264+0000 7f692b9d0700 1 -- 172.21.15.88:0/235040099 wait complete. 2024-03-26T21:22:12.482 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:22:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:22:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.156+0000 7f5451216700 1 -- 172.21.15.88:0/658693518 >> v1:172.21.15.88:6789/0 conn(0x7f544c10ddd0 legacy=0x7f544c110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.156+0000 7f5451216700 1 -- 172.21.15.88:0/658693518 shutdown_connections 2024-03-26T21:22:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.156+0000 7f5451216700 1 -- 172.21.15.88:0/658693518 >> 172.21.15.88:0/658693518 conn(0x7f544c0765e0 msgr2=0x7f544c0769f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.156+0000 7f5451216700 1 -- 172.21.15.88:0/658693518 shutdown_connections 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.156+0000 7f5451216700 1 -- 172.21.15.88:0/658693518 wait complete. 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f5451216700 1 Processor -- start 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f5451216700 1 -- start start 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f5451216700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f544c1a6180 con 0x7f544c101fd0 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f5451216700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f544c1a73a0 con 0x7f544c10ddd0 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f5451216700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f544c1a85c0 con 0x7f544c10a230 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f544a59c700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f544c10a230 0x7f544c1a1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47888/0 (socket says 172.21.15.88:47888) 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f544ad9d700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f544c101fd0 0x7f544c106560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38612/0 (socket says 172.21.15.88:38612) 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.157+0000 7f544a59c700 1 -- 172.21.15.88:0/778544408 learned_addr learned my addr 172.21.15.88:0/778544408 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362773823 0 0) 0x7f544c1a85c0 con 0x7f544c10a230 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54340035b0 con 0x7f544c10a230 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1771136871 0 0) 0x7f544c1a6180 con 0x7f544c101fd0 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f544c1a85c0 con 0x7f544c101fd0 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719891604 0 0) 0x7f544c1a73a0 con 0x7f544c10ddd0 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f544c1a6180 con 0x7f544c10ddd0 2024-03-26T21:22:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2290872863 0 0) 0x7f54340035b0 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f544c1a73a0 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5440001980 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2545260837 0 0) 0x7f544c1a85c0 con 0x7f544c101fd0 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54340035b0 con 0x7f544c101fd0 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5444001b70 con 0x7f544c101fd0 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1113249464 0 0) 0x7f544c1a73a0 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 >> v1:172.21.15.138:6789/0 conn(0x7f544c10ddd0 legacy=0x7f544c1a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 >> v1:172.21.15.88:6789/0 conn(0x7f544c101fd0 legacy=0x7f544c106560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544c1a9820 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f544c1a63b0 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f544c1a6870 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f5440001c80 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.158+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5440005d10 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.159+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.88:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5438000f80 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.159+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 21) ==== 92440+0+0 (unknown 2932806954 0 0) 0x7f544001c7d0 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.160+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f5440089ee0 con 0x7f544c10a230 2024-03-26T21:22:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.160+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.138:6800/2187218446 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f543407faf0 con 0x7f543407bde0 2024-03-26T21:22:13.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.160+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f544008a240 con 0x7f544c10a230 2024-03-26T21:22:13.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.161+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== osd.4 v1:172.21.15.138:6800/2187218446 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f543407faf0 con 0x7f543407bde0 2024-03-26T21:22:13.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.178+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 --> v1:172.21.15.138:6800/2187218446 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5438002ce0 con 0x7f543407bde0 2024-03-26T21:22:13.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5433fff700 1 -- 172.21.15.88:0/778544408 <== osd.4 v1:172.21.15.138:6800/2187218446 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1156102027) 0x7f5438002ce0 con 0x7f543407bde0 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 >> v1:172.21.15.138:6800/2187218446 conn(0x7f543407bde0 legacy=0x7f543407e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 >> v1:172.21.15.88:6800/3939589528 conn(0x7f5434071680 legacy=0x7f5434073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 >> v1:172.21.15.88:6790/0 conn(0x7f544c10a230 legacy=0x7f544c1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f544b59e700 1 -- 172.21.15.88:0/778544408 reap_dead start 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 shutdown_connections 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 >> 172.21.15.88:0/778544408 conn(0x7f544c0765e0 msgr2=0x7f544c1112d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 shutdown_connections 2024-03-26T21:22:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:13.179+0000 7f5451216700 1 -- 172.21.15.88:0/778544408 wait complete. 2024-03-26T21:22:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:12 smithi138 ceph-mon[33706]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:12 smithi138 ceph-mon[33706]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T21:22:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[30075]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[30075]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T21:22:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[39202]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[39202]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T21:22:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:14.063 INFO:teuthology.orchestra.run.smithi088.stdout:107374182435 2024-03-26T21:22:14.063 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi138 to 3760M 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[39202]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi138 to 3760M 2024-03-26T21:22:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:14 smithi088 ceph-mon[30075]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T21:22:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi138 to 3760M 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:14 smithi138 ceph-mon[33706]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T21:22:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.139+0000 7f4b6312e700 1 -- 172.21.15.88:0/3673238051 >> v1:172.21.15.88:6789/0 conn(0x7f4b5c10a1c0 legacy=0x7f4b5c10c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.139+0000 7f4b6312e700 1 -- 172.21.15.88:0/3673238051 shutdown_connections 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.139+0000 7f4b6312e700 1 -- 172.21.15.88:0/3673238051 >> 172.21.15.88:0/3673238051 conn(0x7f4b5c0fcde0 msgr2=0x7f4b5c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.139+0000 7f4b6312e700 1 -- 172.21.15.88:0/3673238051 shutdown_connections 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.139+0000 7f4b6312e700 1 -- 172.21.15.88:0/3673238051 wait complete. 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.139+0000 7f4b6312e700 1 Processor -- start 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.139+0000 7f4b6312e700 1 -- start start 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b6312e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b5c1a6220 con 0x7f4b5c10a1c0 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b6312e700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b5c1a7440 con 0x7f4b5c10dd60 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b6312e700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b5c1a8680 con 0x7f4b5c101530 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b60eca700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4b5c101530 0x7f4b5c1066c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45738/0 (socket says 172.21.15.88:45738) 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b60eca700 1 -- 172.21.15.88:0/1006864197 learned_addr learned my addr 172.21.15.88:0/1006864197 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738854661 0 0) 0x7f4b5c1a6220 con 0x7f4b5c10a1c0 2024-03-26T21:22:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b4c0031f0 con 0x7f4b5c10a1c0 2024-03-26T21:22:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992725740 0 0) 0x7f4b5c1a8680 con 0x7f4b5c101530 2024-03-26T21:22:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b5c1a6220 con 0x7f4b5c101530 2024-03-26T21:22:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3417733414 0 0) 0x7f4b5c1a6220 con 0x7f4b5c101530 2024-03-26T21:22:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b5c1a8680 con 0x7f4b5c101530 2024-03-26T21:22:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307108437 0 0) 0x7f4b5c1a7440 con 0x7f4b5c10dd60 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b5c1a6220 con 0x7f4b5c10dd60 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4b500019e0 con 0x7f4b5c101530 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 587158159 0 0) 0x7f4b4c0031f0 con 0x7f4b5c10a1c0 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b5c1a7440 con 0x7f4b5c10a1c0 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4b48004370 con 0x7f4b5c10a1c0 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3313829686 0 0) 0x7f4b5c1a8680 con 0x7f4b5c101530 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 >> v1:172.21.15.138:6789/0 conn(0x7f4b5c10dd60 legacy=0x7f4b5c1a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 >> v1:172.21.15.88:6789/0 conn(0x7f4b5c10a1c0 legacy=0x7f4b5c1a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b5c1a98c0 con 0x7f4b5c101530 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.140+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b5c1a88b0 con 0x7f4b5c101530 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.141+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f4b50001c70 con 0x7f4b5c101530 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.141+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4b50005ce0 con 0x7f4b5c101530 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.141+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b5c1a8d70 con 0x7f4b5c101530 2024-03-26T21:22:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.141+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4b5c102a10 con 0x7f4b5c101530 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.142+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f4b50003930 con 0x7f4b5c101530 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.142+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f4b5008a050 con 0x7f4b5c101530 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.142+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6809/2517903952 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b48001080 con 0x7f4b4c079f30 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.143+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f4b5008a4c0 con 0x7f4b5c101530 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.143+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== osd.2 v1:172.21.15.88:6809/2517903952 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b48001080 con 0x7f4b4c079f30 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.159+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 --> v1:172.21.15.88:6809/2517903952 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b5c10fba0 con 0x7f4b4c079f30 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b59ffb700 1 -- 172.21.15.88:0/1006864197 <== osd.2 v1:172.21.15.88:6809/2517903952 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1016321414) 0x7f4b5c10fba0 con 0x7f4b4c079f30 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 >> v1:172.21.15.88:6809/2517903952 conn(0x7f4b4c079f30 legacy=0x7f4b4c07c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 >> v1:172.21.15.88:6800/3939589528 conn(0x7f4b4c06f770 legacy=0x7f4b4c071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 >> v1:172.21.15.88:6790/0 conn(0x7f4b5c101530 legacy=0x7f4b5c1066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b616cb700 1 -- 172.21.15.88:0/1006864197 reap_dead start 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 shutdown_connections 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 >> 172.21.15.88:0/1006864197 conn(0x7f4b5c0fcde0 msgr2=0x7f4b5c10e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 shutdown_connections 2024-03-26T21:22:15.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:15.160+0000 7f4b6312e700 1 -- 172.21.15.88:0/1006864197 wait complete. 2024-03-26T21:22:16.062 INFO:teuthology.orchestra.run.smithi088.stdout:77309411368 2024-03-26T21:22:16.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T21:22:16.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:16 smithi088 ceph-mon[30075]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:16.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:16 smithi088 ceph-mon[39202]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:17.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.001+0000 7f2a27f81700 1 -- 172.21.15.88:0/3514826067 >> v1:172.21.15.88:6789/0 conn(0x7f2a2010eeb0 legacy=0x7f2a20111340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:17.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.001+0000 7f2a27f81700 1 -- 172.21.15.88:0/3514826067 shutdown_connections 2024-03-26T21:22:17.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.001+0000 7f2a27f81700 1 -- 172.21.15.88:0/3514826067 >> 172.21.15.88:0/3514826067 conn(0x7f2a200fcde0 msgr2=0x7f2a200ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:17.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.001+0000 7f2a27f81700 1 -- 172.21.15.88:0/3514826067 shutdown_connections 2024-03-26T21:22:17.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.001+0000 7f2a27f81700 1 -- 172.21.15.88:0/3514826067 wait complete. 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a27f81700 1 Processor -- start 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a27f81700 1 -- start start 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a27f81700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2a201a7450 con 0x7f2a2010eeb0 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a27f81700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2a201a8670 con 0x7f2a20103590 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a27f81700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2a201a98b0 con 0x7f2a2010b3a0 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a2551c700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f2a2010b3a0 0x7f2a201a22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45748/0 (socket says 172.21.15.88:45748) 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a2551c700 1 -- 172.21.15.88:0/3753313039 learned_addr learned my addr 172.21.15.88:0/3753313039 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2394960538 0 0) 0x7f2a201a7450 con 0x7f2a2010eeb0 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a0c0035b0 con 0x7f2a2010eeb0 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936874191 0 0) 0x7f2a201a98b0 con 0x7f2a2010b3a0 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a201a7450 con 0x7f2a2010b3a0 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2994176385 0 0) 0x7f2a201a8670 con 0x7f2a20103590 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a201a98b0 con 0x7f2a20103590 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2794741744 0 0) 0x7f2a0c0035b0 con 0x7f2a2010eeb0 2024-03-26T21:22:17.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a201a8670 con 0x7f2a2010eeb0 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.002+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f2a18003bb0 con 0x7f2a2010eeb0 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1479251047 0 0) 0x7f2a201a7450 con 0x7f2a2010b3a0 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a0c0035b0 con 0x7f2a2010b3a0 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f2a10001810 con 0x7f2a2010b3a0 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1480268677 0 0) 0x7f2a201a98b0 con 0x7f2a20103590 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a201a7450 con 0x7f2a20103590 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 347431659 0 0) 0x7f2a201a8670 con 0x7f2a2010eeb0 2024-03-26T21:22:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 >> v1:172.21.15.88:6790/0 conn(0x7f2a2010b3a0 legacy=0x7f2a201a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 >> v1:172.21.15.138:6789/0 conn(0x7f2a20103590 legacy=0x7f2a2010ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a201aaaf0 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a201a7680 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a201a7b40 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f2a18003a00 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.003+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f2a18004fd0 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.004+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2a04000f80 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.004+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f2a1801bb50 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.005+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f2a18088f50 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.005+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.138:6812/1004436505 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a0c07db20 con 0x7f2a0c079e10 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.005+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f2a180893b0 con 0x7f2a2010eeb0 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.006+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== osd.7 v1:172.21.15.138:6812/1004436505 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a0c07db20 con 0x7f2a0c079e10 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.022+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 --> v1:172.21.15.138:6812/1004436505 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a04002ce0 con 0x7f2a0c079e10 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a16ffd700 1 -- 172.21.15.88:0/3753313039 <== osd.7 v1:172.21.15.138:6812/1004436505 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 4172391202) 0x7f2a04002ce0 con 0x7f2a0c079e10 2024-03-26T21:22:17.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 >> v1:172.21.15.138:6812/1004436505 conn(0x7f2a0c079e10 legacy=0x7f2a0c07c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:17.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 >> v1:172.21.15.88:6800/3939589528 conn(0x7f2a0c06f5b0 legacy=0x7f2a0c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:17.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 >> v1:172.21.15.88:6789/0 conn(0x7f2a2010eeb0 legacy=0x7f2a201a5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:17.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a2651e700 1 -- 172.21.15.88:0/3753313039 reap_dead start 2024-03-26T21:22:17.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 shutdown_connections 2024-03-26T21:22:17.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 >> 172.21.15.88:0/3753313039 conn(0x7f2a200fcde0 msgr2=0x7f2a20111300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:17.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 shutdown_connections 2024-03-26T21:22:17.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:17.023+0000 7f2a27f81700 1 -- 172.21.15.88:0/3753313039 wait complete. 2024-03-26T21:22:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:16 smithi138 ceph-mon[33706]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:22:18.565 INFO:teuthology.orchestra.run.smithi088.stdout:128849018910 2024-03-26T21:22:18.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T21:22:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.793+0000 7f3015fa2700 1 -- 172.21.15.88:0/1638002952 >> v1:172.21.15.88:6790/0 conn(0x7f3010101bc0 legacy=0x7f301010f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.793+0000 7f3015fa2700 1 -- 172.21.15.88:0/1638002952 shutdown_connections 2024-03-26T21:22:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.793+0000 7f3015fa2700 1 -- 172.21.15.88:0/1638002952 >> 172.21.15.88:0/1638002952 conn(0x7f30100fcc20 msgr2=0x7f30100ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.793+0000 7f3015fa2700 1 -- 172.21.15.88:0/1638002952 shutdown_connections 2024-03-26T21:22:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.793+0000 7f3015fa2700 1 -- 172.21.15.88:0/1638002952 wait complete. 2024-03-26T21:22:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.794+0000 7f3015fa2700 1 Processor -- start 2024-03-26T21:22:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.794+0000 7f3015fa2700 1 -- start start 2024-03-26T21:22:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.794+0000 7f30157a1700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f3010110ee0 0x7f30101a8370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45754/0 (socket says 172.21.15.88:45754) 2024-03-26T21:22:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.794+0000 7f30157a1700 1 -- 172.21.15.88:0/3750198648 learned_addr learned my addr 172.21.15.88:0/3750198648 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.794+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f301010f8d0 con 0x7f3010101bc0 2024-03-26T21:22:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.794+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f301010fae0 con 0x7f3010100fe0 2024-03-26T21:22:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.794+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30101a9a90 con 0x7f3010110ee0 2024-03-26T21:22:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107206288 0 0) 0x7f301010f8d0 con 0x7f3010101bc0 2024-03-26T21:22:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30000035b0 con 0x7f3010101bc0 2024-03-26T21:22:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162751640 0 0) 0x7f301010fae0 con 0x7f3010100fe0 2024-03-26T21:22:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f301010f8d0 con 0x7f3010100fe0 2024-03-26T21:22:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060301345 0 0) 0x7f30101a9a90 con 0x7f3010110ee0 2024-03-26T21:22:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f301010fae0 con 0x7f3010110ee0 2024-03-26T21:22:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 781711942 0 0) 0x7f30000035b0 con 0x7f3010101bc0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30101a9a90 con 0x7f3010101bc0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4000748245 0 0) 0x7f301010f8d0 con 0x7f3010100fe0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30000035b0 con 0x7f3010100fe0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f2ffc001880 con 0x7f3010101bc0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3004001850 con 0x7f3010100fe0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 824388696 0 0) 0x7f301010fae0 con 0x7f3010110ee0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f301010f8d0 con 0x7f3010110ee0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.795+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3008001c30 con 0x7f3010110ee0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 922782943 0 0) 0x7f301010f8d0 con 0x7f3010110ee0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 >> v1:172.21.15.138:6789/0 conn(0x7f3010100fe0 legacy=0x7f301010c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 >> v1:172.21.15.88:6789/0 conn(0x7f3010101bc0 legacy=0x7f301010ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30101a9c60 con 0x7f3010110ee0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f3008003eb0 con 0x7f3010110ee0 2024-03-26T21:22:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3010103250 con 0x7f3010110ee0 2024-03-26T21:22:18.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f3008006060 con 0x7f3010110ee0 2024-03-26T21:22:18.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.796+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3010103710 con 0x7f3010110ee0 2024-03-26T21:22:18.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.797+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.88:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f301006b0d0 con 0x7f3010110ee0 2024-03-26T21:22:18.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.798+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f300801cc70 con 0x7f3010110ee0 2024-03-26T21:22:18.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.798+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f300808a170 con 0x7f3010110ee0 2024-03-26T21:22:18.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.798+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.138:6808/771295543 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3000086510 con 0x7f3000082800 2024-03-26T21:22:18.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.798+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f300808a5e0 con 0x7f3010110ee0 2024-03-26T21:22:18.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.800+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== osd.6 v1:172.21.15.138:6808/771295543 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3000086510 con 0x7f3000082800 2024-03-26T21:22:18.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.816+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 --> v1:172.21.15.138:6808/771295543 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3010062cd0 con 0x7f3000082800 2024-03-26T21:22:18.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.816+0000 7f300dffb700 1 -- 172.21.15.88:0/3750198648 <== osd.6 v1:172.21.15.138:6808/771295543 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1795465705) 0x7f3010062cd0 con 0x7f3000082800 2024-03-26T21:22:18.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.816+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 >> v1:172.21.15.138:6808/771295543 conn(0x7f3000082800 legacy=0x7f3000084ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:18.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.817+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 >> v1:172.21.15.88:6800/3939589528 conn(0x7f3000077f40 legacy=0x7f300007a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:18.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.817+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 >> v1:172.21.15.88:6790/0 conn(0x7f3010110ee0 legacy=0x7f30101a8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:18.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.817+0000 7f30157a1700 1 -- 172.21.15.88:0/3750198648 reap_dead start 2024-03-26T21:22:18.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.817+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 shutdown_connections 2024-03-26T21:22:18.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.817+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 >> 172.21.15.88:0/3750198648 conn(0x7f30100fcc20 msgr2=0x7f3010110350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:18.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.817+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 shutdown_connections 2024-03-26T21:22:18.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:18.817+0000 7f3015fa2700 1 -- 172.21.15.88:0/3750198648 wait complete. 2024-03-26T21:22:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:18 smithi138 ceph-mon[33706]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T21:22:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:18 smithi088 ceph-mon[30075]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T21:22:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:18 smithi088 ceph-mon[39202]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T21:22:19.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.924+0000 7f4b6ad42700 1 -- 172.21.15.88:0/345133657 >> v1:172.21.15.88:6790/0 conn(0x7f4b64101fa0 legacy=0x7f4b64102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.924+0000 7f4b6ad42700 1 -- 172.21.15.88:0/345133657 shutdown_connections 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.924+0000 7f4b6ad42700 1 -- 172.21.15.88:0/345133657 >> 172.21.15.88:0/345133657 conn(0x7f4b64076630 msgr2=0x7f4b64076a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.924+0000 7f4b6ad42700 1 -- 172.21.15.88:0/345133657 shutdown_connections 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.924+0000 7f4b6ad42700 1 -- 172.21.15.88:0/345133657 wait complete. 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.924+0000 7f4b6ad42700 1 Processor -- start 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.924+0000 7f4b6ad42700 1 -- start start 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b6ad42700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b641a96a0 con 0x7f4b6410a280 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b6ad42700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b641aa8c0 con 0x7f4b64101fa0 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b6ad42700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4b641abb00 con 0x7f4b6410de20 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b692df700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4b6410de20 0x7f4b641a7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45770/0 (socket says 172.21.15.88:45770) 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b692df700 1 -- 172.21.15.88:0/2737968428 learned_addr learned my addr 172.21.15.88:0/2737968428 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802174513 0 0) 0x7f4b641a96a0 con 0x7f4b6410a280 2024-03-26T21:22:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b540035b0 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2333905275 0 0) 0x7f4b540035b0 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b641a96a0 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.925+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4b500010b0 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3377750878 0 0) 0x7f4b641a96a0 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 >> v1:172.21.15.88:6790/0 conn(0x7f4b6410de20 legacy=0x7f4b641a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 >> v1:172.21.15.138:6789/0 conn(0x7f4b64101fa0 legacy=0x7f4b641065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b641acd40 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b641abd30 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f4b50003b10 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4b50004b70 con 0x7f4b6410a280 2024-03-26T21:22:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.928+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b641ac230 con 0x7f4b6410a280 2024-03-26T21:22:19.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.929+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4b48000f80 con 0x7f4b6410a280 2024-03-26T21:22:19.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.929+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f4b5001b670 con 0x7f4b6410a280 2024-03-26T21:22:19.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.930+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f4b50088e50 con 0x7f4b6410a280 2024-03-26T21:22:19.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.930+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6805/698625447 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b5407db80 con 0x7f4b54079e70 2024-03-26T21:22:19.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.930+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f4b50089150 con 0x7f4b6410a280 2024-03-26T21:22:19.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.931+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== osd.1 v1:172.21.15.88:6805/698625447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4b5407db80 con 0x7f4b54079e70 2024-03-26T21:22:19.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.947+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 --> v1:172.21.15.88:6805/698625447 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b48002ce0 con 0x7f4b54079e70 2024-03-26T21:22:19.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b61ffb700 1 -- 172.21.15.88:0/2737968428 <== osd.1 v1:172.21.15.88:6805/698625447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 137719321) 0x7f4b48002ce0 con 0x7f4b54079e70 2024-03-26T21:22:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 >> v1:172.21.15.88:6805/698625447 conn(0x7f4b54079e70 legacy=0x7f4b5407c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 >> v1:172.21.15.88:6800/3939589528 conn(0x7f4b5406f490 legacy=0x7f4b54071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 >> v1:172.21.15.88:6789/0 conn(0x7f4b6410a280 legacy=0x7f4b641a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b692df700 1 -- 172.21.15.88:0/2737968428 reap_dead start 2024-03-26T21:22:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 shutdown_connections 2024-03-26T21:22:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 >> 172.21.15.88:0/2737968428 conn(0x7f4b64076630 msgr2=0x7f4b640ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 shutdown_connections 2024-03-26T21:22:19.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:19.948+0000 7f4b6ad42700 1 -- 172.21.15.88:0/2737968428 wait complete. 2024-03-26T21:22:20.672 INFO:teuthology.orchestra.run.smithi088.stdout:55834574893 2024-03-26T21:22:20.673 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T21:22:20.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:22:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:22:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:20 smithi138 ceph-mon[33706]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T21:22:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:20 smithi088 ceph-mon[30075]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T21:22:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:20 smithi088 ceph-mon[39202]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T21:22:21.365 INFO:teuthology.orchestra.run.smithi088.stdout:231928233999 2024-03-26T21:22:21.365 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T21:22:21.366 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:21.732 INFO:teuthology.orchestra.run.smithi088.stdout:197568495636 2024-03-26T21:22:21.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.189+0000 7f216d6b1700 1 -- 172.21.15.88:0/3932793752 >> v1:172.21.15.88:6790/0 conn(0x7f216810a1c0 legacy=0x7f216810c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.189+0000 7f216d6b1700 1 -- 172.21.15.88:0/3932793752 shutdown_connections 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.189+0000 7f216d6b1700 1 -- 172.21.15.88:0/3932793752 >> 172.21.15.88:0/3932793752 conn(0x7f21680fcde0 msgr2=0x7f21680ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.189+0000 7f216d6b1700 1 -- 172.21.15.88:0/3932793752 shutdown_connections 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.189+0000 7f216d6b1700 1 -- 172.21.15.88:0/3932793752 wait complete. 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.190+0000 7f216d6b1700 1 Processor -- start 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.190+0000 7f216d6b1700 1 -- start start 2024-03-26T21:22:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.190+0000 7f216d6b1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21681a7300 con 0x7f216810a1c0 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.190+0000 7f216d6b1700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21681a8520 con 0x7f2168101530 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.190+0000 7f216d6b1700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21681a9760 con 0x7f216810dd60 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.190+0000 7f21677fe700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f216810dd60 0x7f21681a59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45796/0 (socket says 172.21.15.88:45796) 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.190+0000 7f21677fe700 1 -- 172.21.15.88:0/1182233514 learned_addr learned my addr 172.21.15.88:0/1182233514 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983318386 0 0) 0x7f21681a9760 con 0x7f216810dd60 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21500031f0 con 0x7f216810dd60 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081695448 0 0) 0x7f21681a7300 con 0x7f216810a1c0 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21681a9760 con 0x7f216810a1c0 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041469103 0 0) 0x7f21681a8520 con 0x7f2168101530 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21681a7300 con 0x7f2168101530 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1057142756 0 0) 0x7f21500031f0 con 0x7f216810dd60 2024-03-26T21:22:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21681a8520 con 0x7f216810dd60 2024-03-26T21:22:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f2160001ce0 con 0x7f216810dd60 2024-03-26T21:22:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2583358749 0 0) 0x7f21681a9760 con 0x7f216810a1c0 2024-03-26T21:22:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21500031f0 con 0x7f216810a1c0 2024-03-26T21:22:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f215c001980 con 0x7f216810a1c0 2024-03-26T21:22:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1209974864 0 0) 0x7f21681a7300 con 0x7f2168101530 2024-03-26T21:22:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21681a9760 con 0x7f2168101530 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f2158001830 con 0x7f2168101530 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 379875411 0 0) 0x7f21681a8520 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 >> v1:172.21.15.138:6789/0 conn(0x7f2168101530 legacy=0x7f21681066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 >> v1:172.21.15.88:6789/0 conn(0x7f216810a1c0 legacy=0x7f21681a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21681aa9a0 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21681a7530 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f21681a79f0 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f2160002060 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.191+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f21600060e0 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.192+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.88:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2154000fb0 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.193+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f216001ccc0 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.193+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f216008a180 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.193+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.138:6804/2329820906 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f215007da50 con 0x7f2150079d40 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.193+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f216008a650 con 0x7f216810dd60 2024-03-26T21:22:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.194+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== osd.5 v1:172.21.15.138:6804/2329820906 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f215007da50 con 0x7f2150079d40 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.211+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 --> v1:172.21.15.138:6804/2329820906 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2154002d10 con 0x7f2150079d40 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.211+0000 7f214ffff700 1 -- 172.21.15.88:0/1182233514 <== osd.5 v1:172.21.15.138:6804/2329820906 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 829413231) 0x7f2154002d10 con 0x7f2150079d40 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.211+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 >> v1:172.21.15.138:6804/2329820906 conn(0x7f2150079d40 legacy=0x7f215007c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.212+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 >> v1:172.21.15.88:6800/3939589528 conn(0x7f215006f5e0 legacy=0x7f2150071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.212+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 >> v1:172.21.15.88:6790/0 conn(0x7f216810dd60 legacy=0x7f21681a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.212+0000 7f21677fe700 1 -- 172.21.15.88:0/1182233514 reap_dead start 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.212+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 shutdown_connections 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.212+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 >> 172.21.15.88:0/1182233514 conn(0x7f21680fcde0 msgr2=0x7f2168111260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.212+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 shutdown_connections 2024-03-26T21:22:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:20.212+0000 7f216d6b1700 1 -- 172.21.15.88:0/1182233514 wait complete. 2024-03-26T21:22:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:22.470 INFO:teuthology.orchestra.run.smithi088.stdout:163208757273 2024-03-26T21:22:22.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T21:22:22.686 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:22:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:22:22.974 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:22 smithi088 ceph-mon[39202]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T21:22:22.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:22 smithi088 ceph-mon[30075]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T21:22:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:22 smithi138 ceph-mon[33706]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T21:22:23.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.511+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/1277501294 >> v1:172.21.15.88:6790/0 conn(0x7f4e74108470 legacy=0x7f4e74108850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:23.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.511+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/1277501294 shutdown_connections 2024-03-26T21:22:23.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.511+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/1277501294 >> 172.21.15.88:0/1277501294 conn(0x7f4e740fdef0 msgr2=0x7f4e74100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:23.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.511+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/1277501294 shutdown_connections 2024-03-26T21:22:23.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.511+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/1277501294 wait complete. 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.512+0000 7f4e7c2ce700 1 Processor -- start 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.512+0000 7f4e7c2ce700 1 -- start start 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.512+0000 7f4e7c2ce700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4e741a7230 con 0x7f4e7410ee60 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.512+0000 7f4e7c2ce700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4e741a8450 con 0x7f4e7410b290 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.512+0000 7f4e7c2ce700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4e741a9670 con 0x7f4e74108470 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e7a06a700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4e74108470 0x7f4e7410e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58364/0 (socket says 172.21.15.88:58364) 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e7a06a700 1 -- 172.21.15.88:0/74244277 learned_addr learned my addr 172.21.15.88:0/74244277 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e7a86b700 1 --1- 172.21.15.88:0/74244277 >> v1:172.21.15.88:6789/0 conn(0x7f4e7410ee60 0x7f4e741a5a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39444/0 (socket says 172.21.15.88:39444) 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235943373 0 0) 0x7f4e741a7230 con 0x7f4e7410ee60 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e600035b0 con 0x7f4e7410ee60 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932799380 0 0) 0x7f4e741a9670 con 0x7f4e74108470 2024-03-26T21:22:23.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e741a7230 con 0x7f4e74108470 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524021697 0 0) 0x7f4e741a8450 con 0x7f4e7410b290 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e741a9670 con 0x7f4e7410b290 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3393453725 0 0) 0x7f4e600035b0 con 0x7f4e7410ee60 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e741a8450 con 0x7f4e7410ee60 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4e6c0042d0 con 0x7f4e7410ee60 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1352213613 0 0) 0x7f4e741a7230 con 0x7f4e74108470 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e600035b0 con 0x7f4e74108470 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4e70004510 con 0x7f4e74108470 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1721350741 0 0) 0x7f4e741a9670 con 0x7f4e7410b290 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e741a7230 con 0x7f4e7410b290 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4e64001a30 con 0x7f4e7410b290 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3149765652 0 0) 0x7f4e741a8450 con 0x7f4e7410ee60 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 >> v1:172.21.15.88:6790/0 conn(0x7f4e74108470 legacy=0x7f4e7410e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:23.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 >> v1:172.21.15.138:6789/0 conn(0x7f4e7410b290 legacy=0x7f4e741a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:23.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e741aa8b0 con 0x7f4e7410ee60 2024-03-26T21:22:23.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4e741a98a0 con 0x7f4e7410ee60 2024-03-26T21:22:23.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.513+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4e741a9d60 con 0x7f4e7410ee60 2024-03-26T21:22:23.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.514+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f4e6c0048a0 con 0x7f4e7410ee60 2024-03-26T21:22:23.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.514+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4e6c005ef0 con 0x7f4e7410ee60 2024-03-26T21:22:23.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.514+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4e58000fb0 con 0x7f4e7410ee60 2024-03-26T21:22:23.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.515+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f4e6c001250 con 0x7f4e7410ee60 2024-03-26T21:22:23.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.515+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f4e6c089ff0 con 0x7f4e7410ee60 2024-03-26T21:22:23.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.516+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6801/3845975585 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e6007dc30 con 0x7f4e60079f20 2024-03-26T21:22:23.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.516+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (unknown 2115490125 0 0) 0x7f4e6c08a460 con 0x7f4e7410ee60 2024-03-26T21:22:23.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.516+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== osd.0 v1:172.21.15.88:6801/3845975585 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4e6007dc30 con 0x7f4e60079f20 2024-03-26T21:22:23.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 --> v1:172.21.15.88:6801/3845975585 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4e58002d10 con 0x7f4e60079f20 2024-03-26T21:22:23.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e6b7fe700 1 -- 172.21.15.88:0/74244277 <== osd.0 v1:172.21.15.88:6801/3845975585 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 2240242322) 0x7f4e58002d10 con 0x7f4e60079f20 2024-03-26T21:22:23.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 >> v1:172.21.15.88:6801/3845975585 conn(0x7f4e60079f20 legacy=0x7f4e6007c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 >> v1:172.21.15.88:6800/3939589528 conn(0x7f4e6006f740 legacy=0x7f4e60071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 >> v1:172.21.15.88:6789/0 conn(0x7f4e7410ee60 legacy=0x7f4e741a5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7a86b700 1 -- 172.21.15.88:0/74244277 reap_dead start 2024-03-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 shutdown_connections 2024-03-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 >> 172.21.15.88:0/74244277 conn(0x7f4e740fdef0 msgr2=0x7f4e74111af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 shutdown_connections 2024-03-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:23.533+0000 7f4e7c2ce700 1 -- 172.21.15.88:0/74244277 wait complete. 2024-03-26T21:22:24.318 INFO:teuthology.orchestra.run.smithi088.stdout:34359738420 2024-03-26T21:22:24.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T21:22:24.867 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:24.868 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:24.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:24 smithi088 ceph-mon[39202]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:24.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:24.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:24 smithi088 ceph-mon[30075]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:24.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:25.195 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:24 smithi138 ceph-mon[33706]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:27.175 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:26 smithi138 ceph-mon[33706]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:26 smithi088 ceph-mon[30075]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:26 smithi088 ceph-mon[39202]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.286+0000 7fd29d99c700 1 -- 172.21.15.88:0/2745252288 >> v1:172.21.15.88:6789/0 conn(0x7fd2981035f0 legacy=0x7fd2981039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.286+0000 7fd29d99c700 1 -- 172.21.15.88:0/2745252288 shutdown_connections 2024-03-26T21:22:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.286+0000 7fd29d99c700 1 -- 172.21.15.88:0/2745252288 >> 172.21.15.88:0/2745252288 conn(0x7fd2980fcde0 msgr2=0x7fd2980ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.287+0000 7fd29d99c700 1 -- 172.21.15.88:0/2745252288 shutdown_connections 2024-03-26T21:22:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.287+0000 7fd29d99c700 1 -- 172.21.15.88:0/2745252288 wait complete. 2024-03-26T21:22:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.287+0000 7fd29d99c700 1 Processor -- start 2024-03-26T21:22:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.287+0000 7fd29d99c700 1 -- start start 2024-03-26T21:22:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.287+0000 7fd29d99c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2981a73e0 con 0x7fd29810ef10 2024-03-26T21:22:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.287+0000 7fd29d99c700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2981a8600 con 0x7fd29810b400 2024-03-26T21:22:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.287+0000 7fd29d99c700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2981a9840 con 0x7fd2981035f0 2024-03-26T21:22:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd296ffd700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd2981035f0 0x7fd29810ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58394/0 (socket says 172.21.15.88:58394) 2024-03-26T21:22:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd296ffd700 1 -- 172.21.15.88:0/2297123545 learned_addr learned my addr 172.21.15.88:0/2297123545 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 247868665 0 0) 0x7fd2981a9840 con 0x7fd2981035f0 2024-03-26T21:22:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2800035b0 con 0x7fd2981035f0 2024-03-26T21:22:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794329863 0 0) 0x7fd2981a73e0 con 0x7fd29810ef10 2024-03-26T21:22:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2981a9840 con 0x7fd29810ef10 2024-03-26T21:22:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783131203 0 0) 0x7fd2981a8600 con 0x7fd29810b400 2024-03-26T21:22:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2981a73e0 con 0x7fd29810b400 2024-03-26T21:22:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3205370705 0 0) 0x7fd2981a9840 con 0x7fd29810ef10 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2981a8600 con 0x7fd29810ef10 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd290001470 con 0x7fd29810ef10 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3754103368 0 0) 0x7fd2800035b0 con 0x7fd2981035f0 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2981a9840 con 0x7fd2981035f0 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1975194577 0 0) 0x7fd2981a73e0 con 0x7fd29810b400 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2800035b0 con 0x7fd29810b400 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd288000fe0 con 0x7fd2981035f0 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd28c0017f0 con 0x7fd29810b400 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2677610960 0 0) 0x7fd2981a8600 con 0x7fd29810ef10 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 >> v1:172.21.15.88:6790/0 conn(0x7fd2981035f0 legacy=0x7fd29810ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 >> v1:172.21.15.138:6789/0 conn(0x7fd29810b400 legacy=0x7fd2981a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2981aaa80 con 0x7fd29810ef10 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2981a9a70 con 0x7fd29810ef10 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.288+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2981a9f30 con 0x7fd29810ef10 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.289+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fd290003e90 con 0x7fd29810ef10 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.289+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd290004d10 con 0x7fd29810ef10 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.289+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd284005120 con 0x7fd29810ef10 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.290+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fd290003a00 con 0x7fd29810ef10 2024-03-26T21:22:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.290+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7fd290088ec0 con 0x7fd29810ef10 2024-03-26T21:22:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.295+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fd290056280 con 0x7fd29810ef10 2024-03-26T21:22:28.415 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:28.416 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:28.418 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.521+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd284005410 con 0x7fd29810ef10 2024-03-26T21:22:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.522+0000 7fd29c99a700 1 -- 172.21.15.88:0/2297123545 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (unknown 383520633 0 2838807938) 0x7fd290059f30 con 0x7fd29810ef10 2024-03-26T21:22:28.515 INFO:teuthology.orchestra.run.smithi088.stdout:107374182438 2024-03-26T21:22:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.524+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 >> v1:172.21.15.88:6800/3939589528 conn(0x7fd28006f650 legacy=0x7fd280071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.524+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 >> v1:172.21.15.88:6789/0 conn(0x7fd29810ef10 legacy=0x7fd2981a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.524+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 shutdown_connections 2024-03-26T21:22:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.524+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 >> 172.21.15.88:0/2297123545 conn(0x7fd2980fcde0 msgr2=0x7fd2980ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.524+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 shutdown_connections 2024-03-26T21:22:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:28.524+0000 7fd29d99c700 1 -- 172.21.15.88:0/2297123545 wait complete. 2024-03-26T21:22:29.220 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:28 smithi138 ceph-mon[33706]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:29.220 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2297123545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:22:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:28 smithi088 ceph-mon[39202]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2297123545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:22:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:28 smithi088 ceph-mon[30075]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:22:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2297123545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:22:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2574948772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:22:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2574948772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:22:30.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2574948772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:22:30.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:22:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:22:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:30 smithi088 ceph-mon[30075]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:30 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:30 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:30.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:30 smithi088 ceph-mon[39202]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:30.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:30 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:30.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:30 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:30 smithi138 ceph-mon[33706]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:30 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:30 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:31.347 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182438 for osd.3 2024-03-26T21:22:31.348 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:32.159 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:32.159 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.170+0000 7f94f74f3700 1 -- 172.21.15.88:0/1679973117 >> v1:172.21.15.88:6789/0 conn(0x7f94f010ef70 legacy=0x7f94f0111430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.170+0000 7f94f74f3700 1 -- 172.21.15.88:0/1679973117 shutdown_connections 2024-03-26T21:22:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.170+0000 7f94f74f3700 1 -- 172.21.15.88:0/1679973117 >> 172.21.15.88:0/1679973117 conn(0x7f94f00765d0 msgr2=0x7f94f00769e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.170+0000 7f94f74f3700 1 -- 172.21.15.88:0/1679973117 shutdown_connections 2024-03-26T21:22:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.170+0000 7f94f74f3700 1 -- 172.21.15.88:0/1679973117 wait complete. 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.170+0000 7f94f74f3700 1 Processor -- start 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.170+0000 7f94f74f3700 1 -- start start 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f74f3700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f94f01a7490 con 0x7f94f010b2e0 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f74f3700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f94f01a86b0 con 0x7f94f01073f0 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f74f3700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f94f01a98f0 con 0x7f94f010ef70 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f4a8e700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f94f010b2e0 0x7f94f01a2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39494/0 (socket says 172.21.15.88:39494) 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f4a8e700 1 -- 172.21.15.88:0/1800812384 learned_addr learned my addr 172.21.15.88:0/1800812384 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f5a90700 1 --1- 172.21.15.88:0/1800812384 >> v1:172.21.15.88:6790/0 conn(0x7f94f010ef70 0x7f94f01a5b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58448/0 (socket says 172.21.15.88:58448) 2024-03-26T21:22:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2140853073 0 0) 0x7f94f01a7490 con 0x7f94f010b2e0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94dc0035b0 con 0x7f94f010b2e0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622347342 0 0) 0x7f94f01a98f0 con 0x7f94f010ef70 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f01a7490 con 0x7f94f010ef70 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 956157490 0 0) 0x7f94f01a86b0 con 0x7f94f01073f0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f01a98f0 con 0x7f94f01073f0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1578816585 0 0) 0x7f94dc0035b0 con 0x7f94f010b2e0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94f01a86b0 con 0x7f94f010b2e0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f94e0001970 con 0x7f94f010b2e0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3327021243 0 0) 0x7f94f01a98f0 con 0x7f94f01073f0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94dc0035b0 con 0x7f94f01073f0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f94ec0017f0 con 0x7f94f01073f0 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2075522795 0 0) 0x7f94f01a7490 con 0x7f94f010ef70 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94f01a98f0 con 0x7f94f010ef70 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f94e8004840 con 0x7f94f010ef70 2024-03-26T21:22:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2885747513 0 0) 0x7f94f01a86b0 con 0x7f94f010b2e0 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 >> v1:172.21.15.88:6790/0 conn(0x7f94f010ef70 legacy=0x7f94f01a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 >> v1:172.21.15.138:6789/0 conn(0x7f94f01073f0 legacy=0x7f94f01012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f01aab30 con 0x7f94f010b2e0 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f94f01a76c0 con 0x7f94f010b2e0 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.171+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f94f01a7b80 con 0x7f94f010b2e0 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.172+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f94e0001d20 con 0x7f94f010b2e0 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.172+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f94e0005de0 con 0x7f94f010b2e0 2024-03-26T21:22:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.172+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d4005120 con 0x7f94f010b2e0 2024-03-26T21:22:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.173+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f94e001ca00 con 0x7f94f010b2e0 2024-03-26T21:22:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.173+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f94e0089eb0 con 0x7f94f010b2e0 2024-03-26T21:22:32.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.178+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f94e0057350 con 0x7f94f010b2e0 2024-03-26T21:22:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:32.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.381+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f94d4005410 con 0x7f94f010b2e0 2024-03-26T21:22:32.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.382+0000 7f94e67fc700 1 -- 172.21.15.88:0/1800812384 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (unknown 2131975755 0 154013954) 0x7f94e005b000 con 0x7f94f010b2e0 2024-03-26T21:22:32.375 INFO:teuthology.orchestra.run.smithi088.stdout:163208757276 2024-03-26T21:22:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.384+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 >> v1:172.21.15.88:6800/3939589528 conn(0x7f94dc06f640 legacy=0x7f94dc071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:32.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.384+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 >> v1:172.21.15.88:6789/0 conn(0x7f94f010b2e0 legacy=0x7f94f01a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:32.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.384+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 shutdown_connections 2024-03-26T21:22:32.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.384+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 >> 172.21.15.88:0/1800812384 conn(0x7f94f00765d0 msgr2=0x7f94f01127a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:32.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.384+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 shutdown_connections 2024-03-26T21:22:32.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:32.384+0000 7f94f74f3700 1 -- 172.21.15.88:0/1800812384 wait complete. 2024-03-26T21:22:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:22:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:22:33.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:32 smithi088 ceph-mon[39202]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:33.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1800812384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:22:33.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:32 smithi088 ceph-mon[30075]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:33.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1800812384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:22:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:32 smithi138 ceph-mon[33706]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1800812384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:22:33.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757273 got 163208757276 for osd.5 2024-03-26T21:22:33.573 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.487+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2920161891 >> v1:172.21.15.88:6790/0 conn(0x7fe878108460 legacy=0x7fe878108840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.487+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2920161891 shutdown_connections 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.487+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2920161891 >> 172.21.15.88:0/2920161891 conn(0x7fe8780fdf00 msgr2=0x7fe878100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.487+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2920161891 shutdown_connections 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.487+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2920161891 wait complete. 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.487+0000 7fe87e5e0700 1 Processor -- start 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.487+0000 7fe87e5e0700 1 -- start start 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe87e5e0700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe8781a7480 con 0x7fe87810ee50 2024-03-26T21:22:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe87e5e0700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe8781a86a0 con 0x7fe87810b280 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe87e5e0700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe8781a98e0 con 0x7fe878108460 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe877fff700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fe878108460 0x7fe87810e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58416/0 (socket says 172.21.15.88:58416) 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe877fff700 1 -- 172.21.15.88:0/2574948772 learned_addr learned my addr 172.21.15.88:0/2574948772 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2555451485 0 0) 0x7fe8781a98e0 con 0x7fe878108460 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8600031f0 con 0x7fe878108460 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4145699070 0 0) 0x7fe8781a7480 con 0x7fe87810ee50 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8781a98e0 con 0x7fe87810ee50 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035223792 0 0) 0x7fe8781a86a0 con 0x7fe87810b280 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8781a7480 con 0x7fe87810b280 2024-03-26T21:22:34.039 INFO:teuthology.orchestra.run.smithi088.stdout:128849018913 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3174970545 0 0) 0x7fe8600031f0 con 0x7fe878108460 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8781a86a0 con 0x7fe878108460 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.488+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe868004320 con 0x7fe878108460 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2878102415 0 0) 0x7fe8781a98e0 con 0x7fe87810ee50 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8600031f0 con 0x7fe87810ee50 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe870001590 con 0x7fe87810ee50 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 912849799 0 0) 0x7fe8781a7480 con 0x7fe87810b280 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8781a98e0 con 0x7fe87810b280 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe86c0017f0 con 0x7fe87810b280 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1067233061 0 0) 0x7fe8781a86a0 con 0x7fe878108460 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 >> v1:172.21.15.138:6789/0 conn(0x7fe87810b280 legacy=0x7fe8781a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 >> v1:172.21.15.88:6789/0 conn(0x7fe87810ee50 legacy=0x7fe8781a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8781aab20 con 0x7fe878108460 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe8781a9b10 con 0x7fe878108460 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe8781a9fd0 con 0x7fe878108460 2024-03-26T21:22:34.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fe868001af0 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.489+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fe868004e70 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.490+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe864005120 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.490+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fe868004e50 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.491+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7fe868089e20 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.495+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fe8680571e0 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.695+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fe864005410 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.696+0000 7fe8757fa700 1 -- 172.21.15.88:0/2574948772 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (unknown 1823589948 0 1144977493) 0x7fe86805ae90 con 0x7fe878108460 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.698+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 >> v1:172.21.15.88:6800/3939589528 conn(0x7fe86006f5f0 legacy=0x7fe860071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.698+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 >> v1:172.21.15.88:6790/0 conn(0x7fe878108460 legacy=0x7fe87810e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.698+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 shutdown_connections 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.698+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 >> 172.21.15.88:0/2574948772 conn(0x7fe8780fdf00 msgr2=0x7fe878111ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.698+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 shutdown_connections 2024-03-26T21:22:34.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:29.698+0000 7fe87e5e0700 1 -- 172.21.15.88:0/2574948772 wait complete. 2024-03-26T21:22:34.666 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018913 for osd.4 2024-03-26T21:22:34.666 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:22:35.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:34 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:22:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:22:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:22:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:34 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:22:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi088 to 3760M 2024-03-26T21:22:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-26T21:22:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:22:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:22:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:22:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:35 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi088 to 3760M 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi088 to 3760M 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-26T21:22:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-26T21:22:36.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:22:36.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:22:36.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:36.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:35 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:36 smithi138 ceph-mon[33706]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:22:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:36 smithi138 ceph-mon[33706]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:22:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:36 smithi138 ceph-mon[33706]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:36 smithi138 ceph-mon[33706]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:22:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:36 smithi138 ceph-mon[33706]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:22:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:36 smithi138 ceph-mon[33706]: Deploying daemon alertmanager.a on smithi088 2024-03-26T21:22:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[39202]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:22:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[39202]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:22:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[39202]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[39202]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:22:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[39202]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:22:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[39202]: Deploying daemon alertmanager.a on smithi088 2024-03-26T21:22:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[30075]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:22:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[30075]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:22:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[30075]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[30075]: Updating smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:22:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[30075]: Updating smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:22:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:36 smithi088 ceph-mon[30075]: Deploying daemon alertmanager.a on smithi088 2024-03-26T21:22:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.592+0000 7f9339820700 1 -- 172.21.15.88:0/1824503444 >> v1:172.21.15.88:6789/0 conn(0x7f9334101530 legacy=0x7f9334101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.592+0000 7f9339820700 1 -- 172.21.15.88:0/1824503444 shutdown_connections 2024-03-26T21:22:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.592+0000 7f9339820700 1 -- 172.21.15.88:0/1824503444 >> 172.21.15.88:0/1824503444 conn(0x7f93340fcde0 msgr2=0x7f93340ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.593+0000 7f9339820700 1 -- 172.21.15.88:0/1824503444 shutdown_connections 2024-03-26T21:22:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.593+0000 7f9339820700 1 -- 172.21.15.88:0/1824503444 wait complete. 2024-03-26T21:22:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.593+0000 7f9339820700 1 Processor -- start 2024-03-26T21:22:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.593+0000 7f9339820700 1 -- start start 2024-03-26T21:22:37.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f9339820700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93341a9680 con 0x7f9334101530 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f9339820700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93341aa8c0 con 0x7f933410dd60 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f9339820700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93341abb00 con 0x7f933410a1c0 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f9332ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9334101530 0x7f9334106760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36660/0 (socket says 172.21.15.88:36660) 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f9332ffd700 1 -- 172.21.15.88:0/773803878 learned_addr learned my addr 172.21.15.88:0/773803878 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058709575 0 0) 0x7f93341a9680 con 0x7f9334101530 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f931c0035b0 con 0x7f9334101530 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2749521576 0 0) 0x7f93341abb00 con 0x7f933410a1c0 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93341a9680 con 0x7f933410a1c0 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130521001 0 0) 0x7f93341aa8c0 con 0x7f933410dd60 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93341abb00 con 0x7f933410dd60 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3253010572 0 0) 0x7f931c0035b0 con 0x7f9334101530 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93341aa8c0 con 0x7f9334101530 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f93240045d0 con 0x7f9334101530 2024-03-26T21:22:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1433864786 0 0) 0x7f93341a9680 con 0x7f933410a1c0 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f931c0035b0 con 0x7f933410a1c0 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2214892688 0 0) 0x7f93341abb00 con 0x7f933410dd60 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93341a9680 con 0x7f933410dd60 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9328001230 con 0x7f933410a1c0 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f932c001af0 con 0x7f933410dd60 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3298413266 0 0) 0x7f93341aa8c0 con 0x7f9334101530 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 >> v1:172.21.15.88:6790/0 conn(0x7f933410a1c0 legacy=0x7f93341a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.594+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 >> v1:172.21.15.138:6789/0 conn(0x7f933410dd60 legacy=0x7f93341a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.595+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93341acd40 con 0x7f9334101530 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.595+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93341aaaf0 con 0x7f9334101530 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.595+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f93341aafb0 con 0x7f9334101530 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.595+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f9324001b10 con 0x7f9334101530 2024-03-26T21:22:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.595+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f9324004e70 con 0x7f9334101530 2024-03-26T21:22:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.596+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9320005120 con 0x7f9334101530 2024-03-26T21:22:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.596+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f93240037f0 con 0x7f9334101530 2024-03-26T21:22:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.597+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f9324089ec0 con 0x7f9334101530 2024-03-26T21:22:37.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.601+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f9324057220 con 0x7f9334101530 2024-03-26T21:22:37.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.809+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f9320005410 con 0x7f9334101530 2024-03-26T21:22:37.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.810+0000 7f933881e700 1 -- 172.21.15.88:0/773803878 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (unknown 1274345170 0 3979635220) 0x7f932405aed0 con 0x7f9334101530 2024-03-26T21:22:37.803 INFO:teuthology.orchestra.run.smithi088.stdout:197568495640 2024-03-26T21:22:37.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.812+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 >> v1:172.21.15.88:6800/3939589528 conn(0x7f931c06f740 legacy=0x7f931c071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:37.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.812+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 >> v1:172.21.15.88:6789/0 conn(0x7f9334101530 legacy=0x7f9334106760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:37.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.812+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 shutdown_connections 2024-03-26T21:22:37.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.812+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 >> 172.21.15.88:0/773803878 conn(0x7f93340fcde0 msgr2=0x7f933406a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:37.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.812+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 shutdown_connections 2024-03-26T21:22:37.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.812+0000 7f9339820700 1 -- 172.21.15.88:0/773803878 wait complete. 2024-03-26T21:22:38.147 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/996610234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:22:38.147 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/773803878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:22:38.147 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/996610234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:22:38.147 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/773803878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:22:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/996610234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:22:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/773803878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:22:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.660+0000 7f5e5d331700 1 -- 172.21.15.88:0/3394928993 >> v1:172.21.15.88:6789/0 conn(0x7f5e5810ee80 legacy=0x7f5e58111320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.660+0000 7f5e5d331700 1 -- 172.21.15.88:0/3394928993 shutdown_connections 2024-03-26T21:22:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.660+0000 7f5e5d331700 1 -- 172.21.15.88:0/3394928993 >> 172.21.15.88:0/3394928993 conn(0x7f5e58077770 msgr2=0x7f5e580feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.660+0000 7f5e5d331700 1 -- 172.21.15.88:0/3394928993 shutdown_connections 2024-03-26T21:22:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.660+0000 7f5e5d331700 1 -- 172.21.15.88:0/3394928993 wait complete. 2024-03-26T21:22:38.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.661+0000 7f5e5d331700 1 Processor -- start 2024-03-26T21:22:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.661+0000 7f5e5d331700 1 -- start start 2024-03-26T21:22:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.661+0000 7f5e5d331700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5e581a7490 con 0x7f5e5810ee80 2024-03-26T21:22:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.661+0000 7f5e5d331700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5e581a86b0 con 0x7f5e581084e0 2024-03-26T21:22:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.661+0000 7f5e5d331700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5e581a98f0 con 0x7f5e5810b250 2024-03-26T21:22:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.661+0000 7f5e577fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5e5810ee80 0x7f5e581a5b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36698/0 (socket says 172.21.15.88:36698) 2024-03-26T21:22:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.661+0000 7f5e577fe700 1 -- 172.21.15.88:0/1111882940 learned_addr learned my addr 172.21.15.88:0/1111882940 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:38.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3965530839 0 0) 0x7f5e581a7490 con 0x7f5e5810ee80 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e400035b0 con 0x7f5e5810ee80 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813467354 0 0) 0x7f5e581a98f0 con 0x7f5e5810b250 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e581a7490 con 0x7f5e5810b250 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361334265 0 0) 0x7f5e581a86b0 con 0x7f5e581084e0 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e581a98f0 con 0x7f5e581084e0 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3505069871 0 0) 0x7f5e581a7490 con 0x7f5e5810b250 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e581a86b0 con 0x7f5e5810b250 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3488065888 0 0) 0x7f5e400035b0 con 0x7f5e5810ee80 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e581a7490 con 0x7f5e5810ee80 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5e4c001110 con 0x7f5e5810b250 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5e50004990 con 0x7f5e5810ee80 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3651126824 0 0) 0x7f5e581a98f0 con 0x7f5e581084e0 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e400035b0 con 0x7f5e581084e0 2024-03-26T21:22:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5e480017f0 con 0x7f5e581084e0 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3359233317 0 0) 0x7f5e581a86b0 con 0x7f5e5810b250 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 >> v1:172.21.15.138:6789/0 conn(0x7f5e581084e0 legacy=0x7f5e58107790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 >> v1:172.21.15.88:6789/0 conn(0x7f5e5810ee80 legacy=0x7f5e581a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e581aab30 con 0x7f5e5810b250 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5e581a9b20 con 0x7f5e5810b250 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5e581a9fe0 con 0x7f5e5810b250 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f5e4c003b10 con 0x7f5e5810b250 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.662+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5e4c0049a0 con 0x7f5e5810b250 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.663+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e44005120 con 0x7f5e5810b250 2024-03-26T21:22:38.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.664+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f5e4c003680 con 0x7f5e5810b250 2024-03-26T21:22:38.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.665+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f5e4c088bd0 con 0x7f5e5810b250 2024-03-26T21:22:38.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.669+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f5e4c055f30 con 0x7f5e5810b250 2024-03-26T21:22:38.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.583+0000 7fca37c31700 1 -- 172.21.15.88:0/1194614319 >> v1:172.21.15.88:6789/0 conn(0x7fca3010a1c0 legacy=0x7fca3010c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.583+0000 7fca37c31700 1 -- 172.21.15.88:0/1194614319 shutdown_connections 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.583+0000 7fca37c31700 1 -- 172.21.15.88:0/1194614319 >> 172.21.15.88:0/1194614319 conn(0x7fca300fcde0 msgr2=0x7fca300ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.583+0000 7fca37c31700 1 -- 172.21.15.88:0/1194614319 shutdown_connections 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.583+0000 7fca37c31700 1 -- 172.21.15.88:0/1194614319 wait complete. 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca37c31700 1 Processor -- start 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca37c31700 1 -- start start 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca37c31700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca301adc00 con 0x7fca3010dd60 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca37c31700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca301aede0 con 0x7fca30101530 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca37c31700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca301affc0 con 0x7fca3010a1c0 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca351cc700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fca3010a1c0 0x7fca301a8db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53546/0 (socket says 172.21.15.88:53546) 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca351cc700 1 -- 172.21.15.88:0/300603134 learned_addr learned my addr 172.21.15.88:0/300603134 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2702193060 0 0) 0x7fca301affc0 con 0x7fca3010a1c0 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca1c0035b0 con 0x7fca3010a1c0 2024-03-26T21:22:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432272634 0 0) 0x7fca301adc00 con 0x7fca3010dd60 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca301affc0 con 0x7fca3010dd60 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028409111 0 0) 0x7fca301aede0 con 0x7fca30101530 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca301adc00 con 0x7fca30101530 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3759728308 0 0) 0x7fca1c0035b0 con 0x7fca3010a1c0 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca301aede0 con 0x7fca3010a1c0 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fca2c004610 con 0x7fca3010a1c0 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2862832618 0 0) 0x7fca301affc0 con 0x7fca3010dd60 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.584+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca1c0035b0 con 0x7fca3010dd60 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fca28001bb0 con 0x7fca3010dd60 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 457188147 0 0) 0x7fca301adc00 con 0x7fca30101530 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca301affc0 con 0x7fca30101530 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fca20001250 con 0x7fca30101530 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1877376768 0 0) 0x7fca301aede0 con 0x7fca3010a1c0 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 >> v1:172.21.15.138:6789/0 conn(0x7fca30101530 legacy=0x7fca301066e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 >> v1:172.21.15.88:6789/0 conn(0x7fca3010dd60 legacy=0x7fca301ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca301b11a0 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca301addd0 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fca301ae290 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fca2c000fe0 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.585+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fca2c004d80 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.586+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca14005120 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.587+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fca2c005050 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.587+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7fca2c089d90 con 0x7fca3010a1c0 2024-03-26T21:22:38.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.592+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fca2c057230 con 0x7fca3010a1c0 2024-03-26T21:22:38.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.808+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fca14005410 con 0x7fca3010a1c0 2024-03-26T21:22:38.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.808+0000 7fca26ffd700 1 -- 172.21.15.88:0/300603134 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (unknown 1482059429 0 246087493) 0x7fca2c05aee0 con 0x7fca3010a1c0 2024-03-26T21:22:38.801 INFO:teuthology.orchestra.run.smithi088.stdout:231928234003 2024-03-26T21:22:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.811+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 >> v1:172.21.15.88:6800/3939589528 conn(0x7fca1c06f570 legacy=0x7fca1c071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.811+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 >> v1:172.21.15.88:6790/0 conn(0x7fca3010a1c0 legacy=0x7fca301a8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.812+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 shutdown_connections 2024-03-26T21:22:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.812+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 >> 172.21.15.88:0/300603134 conn(0x7fca300fcde0 msgr2=0x7fca30110a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.812+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 shutdown_connections 2024-03-26T21:22:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.812+0000 7fca37c31700 1 -- 172.21.15.88:0/300603134 wait complete. 2024-03-26T21:22:38.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.886+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5e44005410 con 0x7f5e5810b250 2024-03-26T21:22:38.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.886+0000 7f5e3ffff700 1 -- 172.21.15.88:0/1111882940 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1301045289) 0x7f5e4c059be0 con 0x7f5e5810b250 2024-03-26T21:22:38.880 INFO:teuthology.orchestra.run.smithi088.stdout:55834574897 2024-03-26T21:22:38.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.888+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 >> v1:172.21.15.88:6800/3939589528 conn(0x7f5e4006f660 legacy=0x7f5e40071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.888+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 >> v1:172.21.15.88:6790/0 conn(0x7f5e5810b250 legacy=0x7f5e581a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:38.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.888+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 shutdown_connections 2024-03-26T21:22:38.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.888+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 >> 172.21.15.88:0/1111882940 conn(0x7f5e58077770 msgr2=0x7f5e581112e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:38.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.889+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 shutdown_connections 2024-03-26T21:22:38.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:38.889+0000 7f5e5d331700 1 -- 172.21.15.88:0/1111882940 wait complete. 2024-03-26T21:22:38.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:38 smithi138 ceph-mon[33706]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:39.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:38 smithi088 ceph-mon[30075]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:39.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/300603134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:22:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1111882940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:22:39.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:38 smithi088 ceph-mon[39202]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:39.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/300603134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:22:39.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1111882940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:22:39.289 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/300603134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:22:39.289 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1111882940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:22:39.367 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495640 for osd.6 2024-03-26T21:22:39.367 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:39.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.804+0000 7fd5c28e1700 1 -- 172.21.15.88:0/2530094313 >> v1:172.21.15.88:6789/0 conn(0x7fd5bc101530 legacy=0x7fd5bc101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.804+0000 7fd5c28e1700 1 -- 172.21.15.88:0/2530094313 shutdown_connections 2024-03-26T21:22:39.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.804+0000 7fd5c28e1700 1 -- 172.21.15.88:0/2530094313 >> 172.21.15.88:0/2530094313 conn(0x7fd5bc0fcde0 msgr2=0x7fd5bc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:39.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.804+0000 7fd5c28e1700 1 -- 172.21.15.88:0/2530094313 shutdown_connections 2024-03-26T21:22:39.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.804+0000 7fd5c28e1700 1 -- 172.21.15.88:0/2530094313 wait complete. 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5c28e1700 1 Processor -- start 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5c28e1700 1 -- start start 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5c28e1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd5bc1a7430 con 0x7fd5bc10dd60 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5c28e1700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd5bc1a8650 con 0x7fd5bc101530 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5c28e1700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd5bc1a9890 con 0x7fd5bc10a1c0 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5bb7fe700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd5bc10a1c0 0x7fd5bc1a21e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53582/0 (socket says 172.21.15.88:53582) 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5bb7fe700 1 -- 172.21.15.88:0/100879336 learned_addr learned my addr 172.21.15.88:0/100879336 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2385080181 0 0) 0x7fd5bc1a7430 con 0x7fd5bc10dd60 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5a40031f0 con 0x7fd5bc10dd60 2024-03-26T21:22:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3282339247 0 0) 0x7fd5bc1a9890 con 0x7fd5bc10a1c0 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5bc1a7430 con 0x7fd5bc10a1c0 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377038816 0 0) 0x7fd5bc1a8650 con 0x7fd5bc101530 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.805+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5bc1a9890 con 0x7fd5bc101530 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2253592700 0 0) 0x7fd5a40031f0 con 0x7fd5bc10dd60 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5bc1a8650 con 0x7fd5bc10dd60 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd5b4001590 con 0x7fd5bc10dd60 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2967029495 0 0) 0x7fd5bc1a9890 con 0x7fd5bc101530 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5a40031f0 con 0x7fd5bc101530 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2024904730 0 0) 0x7fd5bc1a7430 con 0x7fd5bc10a1c0 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5bc1a9890 con 0x7fd5bc10a1c0 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd5ac001950 con 0x7fd5bc101530 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd5b00017f0 con 0x7fd5bc10a1c0 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3421182761 0 0) 0x7fd5bc1a8650 con 0x7fd5bc10dd60 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 >> v1:172.21.15.88:6790/0 conn(0x7fd5bc10a1c0 legacy=0x7fd5bc1a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 >> v1:172.21.15.138:6789/0 conn(0x7fd5bc101530 legacy=0x7fd5bc106890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5bc1aaad0 con 0x7fd5bc10dd60 2024-03-26T21:22:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd5bc1a9ac0 con 0x7fd5bc10dd60 2024-03-26T21:22:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fd5b4003e90 con 0x7fd5bc10dd60 2024-03-26T21:22:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.806+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fd5b4004df0 con 0x7fd5bc10dd60 2024-03-26T21:22:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.807+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5bc1a9f80 con 0x7fd5bc10dd60 2024-03-26T21:22:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.807+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fd5b401b9d0 con 0x7fd5bc10dd60 2024-03-26T21:22:39.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.807+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5bc102a10 con 0x7fd5bc10dd60 2024-03-26T21:22:39.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.808+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7fd5b4089030 con 0x7fd5bc10dd60 2024-03-26T21:22:39.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:39.813+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fd5b4056390 con 0x7fd5bc10dd60 2024-03-26T21:22:39.895 INFO:teuthology.orchestra.run.smithi088.stdout:77309411373 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.479+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/3699249581 >> v1:172.21.15.88:6789/0 conn(0x7f8c341035d0 legacy=0x7f8c341039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.479+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/3699249581 shutdown_connections 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.479+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/3699249581 >> 172.21.15.88:0/3699249581 conn(0x7f8c340fcde0 msgr2=0x7f8c340ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.479+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/3699249581 shutdown_connections 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.479+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/3699249581 wait complete. 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.479+0000 7f8c3a7ed700 1 Processor -- start 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c3a7ed700 1 -- start start 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c3a7ed700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8c341a7420 con 0x7f8c341035d0 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c3a7ed700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8c341a8640 con 0x7f8c3410eef0 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c3a7ed700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8c341a9880 con 0x7f8c3410b3e0 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c33fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8c341035d0 0x7f8c3410ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36640/0 (socket says 172.21.15.88:36640) 2024-03-26T21:22:39.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c33fff700 1 -- 172.21.15.88:0/996610234 learned_addr learned my addr 172.21.15.88:0/996610234 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212886588 0 0) 0x7f8c341a9880 con 0x7f8c3410b3e0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c1c0031f0 con 0x7f8c3410b3e0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189606491 0 0) 0x7f8c341a7420 con 0x7f8c341035d0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c341a9880 con 0x7f8c341035d0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407502347 0 0) 0x7f8c341a8640 con 0x7f8c3410eef0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.480+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c341a7420 con 0x7f8c3410eef0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2082245051 0 0) 0x7f8c1c0031f0 con 0x7f8c3410b3e0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c341a8640 con 0x7f8c3410b3e0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f8c280013d0 con 0x7f8c3410b3e0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3121266893 0 0) 0x7f8c341a9880 con 0x7f8c341035d0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c1c0031f0 con 0x7f8c341035d0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1837476284 0 0) 0x7f8c341a7420 con 0x7f8c3410eef0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c341a9880 con 0x7f8c3410eef0 2024-03-26T21:22:39.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f8c240045d0 con 0x7f8c341035d0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f8c2c001af0 con 0x7f8c3410eef0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 171169844 0 0) 0x7f8c341a8640 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 >> v1:172.21.15.138:6789/0 conn(0x7f8c3410eef0 legacy=0x7f8c341a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 >> v1:172.21.15.88:6789/0 conn(0x7f8c341035d0 legacy=0x7f8c3410ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c341aaac0 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8c341a7650 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8c341a7b10 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f8c28001680 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.481+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f8c280049d0 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.482+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c20005120 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.482+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f8c2801b330 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.483+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f8c28088a00 con 0x7f8c3410b3e0 2024-03-26T21:22:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.487+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f8c28055dc0 con 0x7f8c3410b3e0 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.722+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8c20005410 con 0x7f8c3410b3e0 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.722+0000 7f8c317fa700 1 -- 172.21.15.88:0/996610234 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 3275464226) 0x7f8c28059a70 con 0x7f8c3410b3e0 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.724+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 >> v1:172.21.15.88:6800/3939589528 conn(0x7f8c1c06f6d0 legacy=0x7f8c1c071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.724+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 >> v1:172.21.15.88:6790/0 conn(0x7f8c3410b3e0 legacy=0x7f8c341a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.724+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 shutdown_connections 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.724+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 >> 172.21.15.88:0/996610234 conn(0x7f8c340fcde0 msgr2=0x7f8c340ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.724+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 shutdown_connections 2024-03-26T21:22:39.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:37.724+0000 7f8c3a7ed700 1 -- 172.21.15.88:0/996610234 wait complete. 2024-03-26T21:22:40.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.021+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd5bc105f00 con 0x7fd5bc10dd60 2024-03-26T21:22:40.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.021+0000 7fd5b97fa700 1 -- 172.21.15.88:0/100879336 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 739478307) 0x7fd5b405a040 con 0x7fd5bc10dd60 2024-03-26T21:22:40.014 INFO:teuthology.orchestra.run.smithi088.stdout:34359738423 2024-03-26T21:22:40.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.023+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 >> v1:172.21.15.88:6800/3939589528 conn(0x7fd5a4073af0 legacy=0x7fd5a4075fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:40.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.023+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 >> v1:172.21.15.88:6789/0 conn(0x7fd5bc10dd60 legacy=0x7fd5bc1a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:40.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.023+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 shutdown_connections 2024-03-26T21:22:40.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.023+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 >> 172.21.15.88:0/100879336 conn(0x7fd5bc0fcde0 msgr2=0x7fd5bc1101c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:40.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.023+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 shutdown_connections 2024-03-26T21:22:40.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:40.023+0000 7fd5c28e1700 1 -- 172.21.15.88:0/100879336 wait complete. 2024-03-26T21:22:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:39 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:39 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:40.407 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:39 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:40.697 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411373 for osd.2 2024-03-26T21:22:40.697 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:40.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:22:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:22:41.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233999 got 231928234003 for osd.7 2024-03-26T21:22:41.191 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:40 smithi088 ceph-mon[39202]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/100879336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:22:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:40 smithi088 ceph-mon[30075]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/100879336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:22:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:40 smithi138 ceph-mon[33706]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/100879336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:22:42.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574897 for osd.1 2024-03-26T21:22:42.163 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:42.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:42.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:42.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:22:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:22:43.253 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:42 smithi088 ceph-mon[30075]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:43.253 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:42 smithi088 ceph-mon[39202]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:43.279 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738423 for osd.0 2024-03-26T21:22:43.279 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:43.279 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T21:22:43.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:22:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:42 smithi138 ceph-mon[33706]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:43.525 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:45.292 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:44 smithi088 ceph-mon[30075]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:45.292 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:45 smithi088 ceph-mon[39202]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:44 smithi138 ceph-mon[33706]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.947+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2574685550 >> v1:172.21.15.88:6790/0 conn(0x7f52b810b290 legacy=0x7f52b810d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.947+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2574685550 shutdown_connections 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.947+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2574685550 >> 172.21.15.88:0/2574685550 conn(0x7f52b80fdef0 msgr2=0x7f52b8100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.947+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2574685550 shutdown_connections 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.947+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2574685550 wait complete. 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52bdbf1700 1 Processor -- start 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52bdbf1700 1 -- start start 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52bdbf1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52b81a74d0 con 0x7f52b810b290 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52bdbf1700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52b81a86f0 con 0x7f52b810ee60 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52bdbf1700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52b81a9930 con 0x7f52b8108470 2024-03-26T21:22:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52b77fe700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f52b8108470 0x7f52b810e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:57454/0 (socket says 172.21.15.88:57454) 2024-03-26T21:22:45.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52b77fe700 1 -- 172.21.15.88:0/2153052322 learned_addr learned my addr 172.21.15.88:0/2153052322 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:45.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.948+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104343115 0 0) 0x7f52b81a9930 con 0x7f52b8108470 2024-03-26T21:22:45.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52a00035b0 con 0x7f52b8108470 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916620605 0 0) 0x7f52b81a74d0 con 0x7f52b810b290 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52b81a9930 con 0x7f52b810b290 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1045585558 0 0) 0x7f52b81a86f0 con 0x7f52b810ee60 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52b81a74d0 con 0x7f52b810ee60 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2312408837 0 0) 0x7f52a00035b0 con 0x7f52b8108470 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52b81a86f0 con 0x7f52b8108470 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f52a8001930 con 0x7f52b8108470 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 417328393 0 0) 0x7f52b81a9930 con 0x7f52b810b290 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52a00035b0 con 0x7f52b810b290 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f52ac004430 con 0x7f52b810b290 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3391536225 0 0) 0x7f52b81a74d0 con 0x7f52b810ee60 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52b81a9930 con 0x7f52b810ee60 2024-03-26T21:22:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f52b0001590 con 0x7f52b810ee60 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1142731433 0 0) 0x7f52b81a86f0 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 >> v1:172.21.15.138:6789/0 conn(0x7f52b810ee60 legacy=0x7f52b81a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 >> v1:172.21.15.88:6789/0 conn(0x7f52b810b290 legacy=0x7f52b81a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b81aab70 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52b81a7700 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.949+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f52b81a7bc0 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.950+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f52a8001d00 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.950+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f52a8005ce0 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.950+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a4005150 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.951+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f52a801c640 con 0x7f52b8108470 2024-03-26T21:22:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.951+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f52a8089c40 con 0x7f52b8108470 2024-03-26T21:22:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:45.956+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f52a80570e0 con 0x7f52b8108470 2024-03-26T21:22:46.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.167+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 --> v1:172.21.15.88:6800/3939589528 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f52a4000bc0 con 0x7f52a006f5d0 2024-03-26T21:22:46.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.177+0000 7f52b4ff9700 1 -- 172.21.15.88:0/2153052322 <== mgr.14622 v1:172.21.15.88:6800/3939589528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346463 (unknown 2965378022 0 2619287153) 0x7f52a4000bc0 con 0x7f52a006f5d0 2024-03-26T21:22:46.176 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:22:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.181+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 >> v1:172.21.15.88:6800/3939589528 conn(0x7f52a006f5d0 legacy=0x7f52a0071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.181+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 >> v1:172.21.15.88:6790/0 conn(0x7f52b8108470 legacy=0x7f52b810e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.181+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 shutdown_connections 2024-03-26T21:22:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.181+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 >> 172.21.15.88:0/2153052322 conn(0x7f52b80fdef0 msgr2=0x7f52b8111af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.181+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 shutdown_connections 2024-03-26T21:22:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:46.181+0000 7f52bdbf1700 1 -- 172.21.15.88:0/2153052322 wait complete. 2024-03-26T21:22:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-03-26T21:22:47.001 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-26T21:22:45.391901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1008,"num_read_kb":854,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235932,"kb_used_data":7292,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725284,"statfs":{"total":767960285184,"available":767718690816,"internally_reserved":0,"allocated":7467008,"data_stored":3991600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002746"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374766+0000","last_change":"2024-03-26T21:21:31.349104+0000","last_active":"2024-03-26T21:22:09.374766+0000","last_peered":"2024-03-26T21:22:09.374766+0000","last_clean":"2024-03-26T21:22:09.374766+0000","last_became_active":"2024-03-26T21:21:31.348758+0000","last_became_peered":"2024-03-26T21:21:31.348758+0000","last_unstale":"2024-03-26T21:22:09.374766+0000","last_undegraded":"2024-03-26T21:22:09.374766+0000","last_fullsized":"2024-03-26T21:22:09.374766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:32:54.707488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375349+0000","last_change":"2024-03-26T21:21:29.206796+0000","last_active":"2024-03-26T21:22:09.375349+0000","last_peered":"2024-03-26T21:22:09.375349+0000","last_clean":"2024-03-26T21:22:09.375349+0000","last_became_active":"2024-03-26T21:21:29.206440+0000","last_became_peered":"2024-03-26T21:21:29.206440+0000","last_unstale":"2024-03-26T21:22:09.375349+0000","last_undegraded":"2024-03-26T21:22:09.375349+0000","last_fullsized":"2024-03-26T21:22:09.375349+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:03:26.449042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494434+0000","last_change":"2024-03-26T21:21:25.189507+0000","last_active":"2024-03-26T21:22:09.494434+0000","last_peered":"2024-03-26T21:22:09.494434+0000","last_clean":"2024-03-26T21:22:09.494434+0000","last_became_active":"2024-03-26T21:21:25.189192+0000","last_became_peered":"2024-03-26T21:21:25.189192+0000","last_unstale":"2024-03-26T21:22:09.494434+0000","last_undegraded":"2024-03-26T21:22:09.494434+0000","last_fullsized":"2024-03-26T21:22:09.494434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:24:36.169921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374670+0000","last_change":"2024-03-26T21:21:27.201221+0000","last_active":"2024-03-26T21:22:09.374670+0000","last_peered":"2024-03-26T21:22:09.374670+0000","last_clean":"2024-03-26T21:22:09.374670+0000","last_became_active":"2024-03-26T21:21:27.200913+0000","last_became_peered":"2024-03-26T21:21:27.200913+0000","last_unstale":"2024-03-26T21:22:09.374670+0000","last_undegraded":"2024-03-26T21:22:09.374670+0000","last_fullsized":"2024-03-26T21:22:09.374670+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:39:17.901410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375532+0000","last_change":"2024-03-26T21:21:31.216461+0000","last_active":"2024-03-26T21:22:09.375532+0000","last_peered":"2024-03-26T21:22:09.375532+0000","last_clean":"2024-03-26T21:22:09.375532+0000","last_became_active":"2024-03-26T21:21:31.216128+0000","last_became_peered":"2024-03-26T21:21:31.216128+0000","last_unstale":"2024-03-26T21:22:09.375532+0000","last_undegraded":"2024-03-26T21:22:09.375532+0000","last_fullsized":"2024-03-26T21:22:09.375532+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:47:19.990154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376376+0000","last_change":"2024-03-26T21:21:29.206305+0000","last_active":"2024-03-26T21:22:09.376376+0000","last_peered":"2024-03-26T21:22:09.376376+0000","last_clean":"2024-03-26T21:22:09.376376+0000","last_became_active":"2024-03-26T21:21:29.205975+0000","last_became_peered":"2024-03-26T21:21:29.205975+0000","last_unstale":"2024-03-26T21:22:09.376376+0000","last_undegraded":"2024-03-26T21:22:09.376376+0000","last_fullsized":"2024-03-26T21:22:09.376376+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:26:09.381171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375686+0000","last_change":"2024-03-26T21:21:25.189428+0000","last_active":"2024-03-26T21:22:09.375686+0000","last_peered":"2024-03-26T21:22:09.375686+0000","last_clean":"2024-03-26T21:22:09.375686+0000","last_became_active":"2024-03-26T21:21:25.189152+0000","last_became_peered":"2024-03-26T21:21:25.189152+0000","last_unstale":"2024-03-26T21:22:09.375686+0000","last_undegraded":"2024-03-26T21:22:09.375686+0000","last_fullsized":"2024-03-26T21:22:09.375686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:57:13.834170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375492+0000","last_change":"2024-03-26T21:21:27.206416+0000","last_active":"2024-03-26T21:22:09.375492+0000","last_peered":"2024-03-26T21:22:09.375492+0000","last_clean":"2024-03-26T21:22:09.375492+0000","last_became_active":"2024-03-26T21:21:27.206108+0000","last_became_peered":"2024-03-26T21:21:27.206108+0000","last_unstale":"2024-03-26T21:22:09.375492+0000","last_undegraded":"2024-03-26T21:22:09.375492+0000","last_fullsized":"2024-03-26T21:22:09.375492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:16:20.138623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838517+0000","last_change":"2024-03-26T21:21:31.215674+0000","last_active":"2024-03-26T21:22:09.838517+0000","last_peered":"2024-03-26T21:22:09.838517+0000","last_clean":"2024-03-26T21:22:09.838517+0000","last_became_active":"2024-03-26T21:21:31.215409+0000","last_became_peered":"2024-03-26T21:21:31.215409+0000","last_unstale":"2024-03-26T21:22:09.838517+0000","last_undegraded":"2024-03-26T21:22:09.838517+0000","last_fullsized":"2024-03-26T21:22:09.838517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:36:38.087934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840290+0000","last_change":"2024-03-26T21:21:29.204643+0000","last_active":"2024-03-26T21:22:09.840290+0000","last_peered":"2024-03-26T21:22:09.840290+0000","last_clean":"2024-03-26T21:22:09.840290+0000","last_became_active":"2024-03-26T21:21:29.204343+0000","last_became_peered":"2024-03-26T21:21:29.204343+0000","last_unstale":"2024-03-26T21:22:09.840290+0000","last_undegraded":"2024-03-26T21:22:09.840290+0000","last_fullsized":"2024-03-26T21:22:09.840290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:39:36.717265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840336+0000","last_change":"2024-03-26T21:21:25.190623+0000","last_active":"2024-03-26T21:22:09.840336+0000","last_peered":"2024-03-26T21:22:09.840336+0000","last_clean":"2024-03-26T21:22:09.840336+0000","last_became_active":"2024-03-26T21:21:25.190084+0000","last_became_peered":"2024-03-26T21:21:25.190084+0000","last_unstale":"2024-03-26T21:22:09.840336+0000","last_undegraded":"2024-03-26T21:22:09.840336+0000","last_fullsized":"2024-03-26T21:22:09.840336+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:24:42.314966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376172+0000","last_change":"2024-03-26T21:21:48.190967+0000","last_active":"2024-03-26T21:22:09.376172+0000","last_peered":"2024-03-26T21:22:09.376172+0000","last_clean":"2024-03-26T21:22:09.376172+0000","last_became_active":"2024-03-26T21:21:48.177524+0000","last_became_peered":"2024-03-26T21:21:48.177524+0000","last_unstale":"2024-03-26T21:22:09.376172+0000","last_undegraded":"2024-03-26T21:22:09.376172+0000","last_fullsized":"2024-03-26T21:22:09.376172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:33:07.643016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494368+0000","last_change":"2024-03-26T21:21:31.219439+0000","last_active":"2024-03-26T21:22:09.494368+0000","last_peered":"2024-03-26T21:22:09.494368+0000","last_clean":"2024-03-26T21:22:09.494368+0000","last_became_active":"2024-03-26T21:21:31.219016+0000","last_became_peered":"2024-03-26T21:21:31.219016+0000","last_unstale":"2024-03-26T21:22:09.494368+0000","last_undegraded":"2024-03-26T21:22:09.494368+0000","last_fullsized":"2024-03-26T21:22:09.494368+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:04:29.242885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840018+0000","last_change":"2024-03-26T21:21:29.197106+0000","last_active":"2024-03-26T21:22:09.840018+0000","last_peered":"2024-03-26T21:22:09.840018+0000","last_clean":"2024-03-26T21:22:09.840018+0000","last_became_active":"2024-03-26T21:21:29.196780+0000","last_became_peered":"2024-03-26T21:21:29.196780+0000","last_unstale":"2024-03-26T21:22:09.840018+0000","last_undegraded":"2024-03-26T21:22:09.840018+0000","last_fullsized":"2024-03-26T21:22:09.840018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:34:33.394076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839675+0000","last_change":"2024-03-26T21:21:25.193368+0000","last_active":"2024-03-26T21:22:09.839675+0000","last_peered":"2024-03-26T21:22:09.839675+0000","last_clean":"2024-03-26T21:22:09.839675+0000","last_became_active":"2024-03-26T21:21:25.193073+0000","last_became_peered":"2024-03-26T21:21:25.193073+0000","last_unstale":"2024-03-26T21:22:09.839675+0000","last_undegraded":"2024-03-26T21:22:09.839675+0000","last_fullsized":"2024-03-26T21:22:09.839675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:30.706559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375554+0000","last_change":"2024-03-26T21:21:27.208458+0000","last_active":"2024-03-26T21:22:09.375554+0000","last_peered":"2024-03-26T21:22:09.375554+0000","last_clean":"2024-03-26T21:22:09.375554+0000","last_became_active":"2024-03-26T21:21:27.208126+0000","last_became_peered":"2024-03-26T21:21:27.208126+0000","last_unstale":"2024-03-26T21:22:09.375554+0000","last_undegraded":"2024-03-26T21:22:09.375554+0000","last_fullsized":"2024-03-26T21:22:09.375554+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:47:19.052796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376161+0000","last_change":"2024-03-26T21:21:31.349018+0000","last_active":"2024-03-26T21:22:09.376161+0000","last_peered":"2024-03-26T21:22:09.376161+0000","last_clean":"2024-03-26T21:22:09.376161+0000","last_became_active":"2024-03-26T21:21:31.348709+0000","last_became_peered":"2024-03-26T21:21:31.348709+0000","last_unstale":"2024-03-26T21:22:09.376161+0000","last_undegraded":"2024-03-26T21:22:09.376161+0000","last_fullsized":"2024-03-26T21:22:09.376161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:04:23.573573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375942+0000","last_change":"2024-03-26T21:21:29.203037+0000","last_active":"2024-03-26T21:22:09.375942+0000","last_peered":"2024-03-26T21:22:09.375942+0000","last_clean":"2024-03-26T21:22:09.375942+0000","last_became_active":"2024-03-26T21:21:29.202728+0000","last_became_peered":"2024-03-26T21:21:29.202728+0000","last_unstale":"2024-03-26T21:22:09.375942+0000","last_undegraded":"2024-03-26T21:22:09.375942+0000","last_fullsized":"2024-03-26T21:22:09.375942+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:49:46.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375235+0000","last_change":"2024-03-26T21:21:25.188941+0000","last_active":"2024-03-26T21:22:09.375235+0000","last_peered":"2024-03-26T21:22:09.375235+0000","last_clean":"2024-03-26T21:22:09.375235+0000","last_became_active":"2024-03-26T21:21:25.188559+0000","last_became_peered":"2024-03-26T21:21:25.188559+0000","last_unstale":"2024-03-26T21:22:09.375235+0000","last_undegraded":"2024-03-26T21:22:09.375235+0000","last_fullsized":"2024-03-26T21:22:09.375235+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:46:08.987380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375163+0000","last_change":"2024-03-26T21:21:27.202424+0000","last_active":"2024-03-26T21:22:09.375163+0000","last_peered":"2024-03-26T21:22:09.375163+0000","last_clean":"2024-03-26T21:22:09.375163+0000","last_became_active":"2024-03-26T21:21:27.202113+0000","last_became_peered":"2024-03-26T21:21:27.202113+0000","last_unstale":"2024-03-26T21:22:09.375163+0000","last_undegraded":"2024-03-26T21:22:09.375163+0000","last_fullsized":"2024-03-26T21:22:09.375163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:25:50.165698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376528+0000","last_change":"2024-03-26T21:21:31.348441+0000","last_active":"2024-03-26T21:22:09.376528+0000","last_peered":"2024-03-26T21:22:09.376528+0000","last_clean":"2024-03-26T21:22:09.376528+0000","last_became_active":"2024-03-26T21:21:31.348124+0000","last_became_peered":"2024-03-26T21:21:31.348124+0000","last_unstale":"2024-03-26T21:22:09.376528+0000","last_undegraded":"2024-03-26T21:22:09.376528+0000","last_fullsized":"2024-03-26T21:22:09.376528+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:48:58.772269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378559+0000","last_change":"2024-03-26T21:21:29.207591+0000","last_active":"2024-03-26T21:22:09.378559+0000","last_peered":"2024-03-26T21:22:09.378559+0000","last_clean":"2024-03-26T21:22:09.378559+0000","last_became_active":"2024-03-26T21:21:29.207132+0000","last_became_peered":"2024-03-26T21:21:29.207132+0000","last_unstale":"2024-03-26T21:22:09.378559+0000","last_undegraded":"2024-03-26T21:22:09.378559+0000","last_fullsized":"2024-03-26T21:22:09.378559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:41:23.529202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839178+0000","last_change":"2024-03-26T21:21:25.190870+0000","last_active":"2024-03-26T21:22:09.839178+0000","last_peered":"2024-03-26T21:22:09.839178+0000","last_clean":"2024-03-26T21:22:09.839178+0000","last_became_active":"2024-03-26T21:21:25.190522+0000","last_became_peered":"2024-03-26T21:21:25.190522+0000","last_unstale":"2024-03-26T21:22:09.839178+0000","last_undegraded":"2024-03-26T21:22:09.839178+0000","last_fullsized":"2024-03-26T21:22:09.839178+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:10:37.983490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838892+0000","last_change":"2024-03-26T21:21:27.203220+0000","last_active":"2024-03-26T21:22:09.838892+0000","last_peered":"2024-03-26T21:22:09.838892+0000","last_clean":"2024-03-26T21:22:09.838892+0000","last_became_active":"2024-03-26T21:21:27.202842+0000","last_became_peered":"2024-03-26T21:21:27.202842+0000","last_unstale":"2024-03-26T21:22:09.838892+0000","last_undegraded":"2024-03-26T21:22:09.838892+0000","last_fullsized":"2024-03-26T21:22:09.838892+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:44:39.557554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378097+0000","last_change":"2024-03-26T21:21:31.218162+0000","last_active":"2024-03-26T21:22:09.378097+0000","last_peered":"2024-03-26T21:22:09.378097+0000","last_clean":"2024-03-26T21:22:09.378097+0000","last_became_active":"2024-03-26T21:21:31.217319+0000","last_became_peered":"2024-03-26T21:21:31.217319+0000","last_unstale":"2024-03-26T21:22:09.378097+0000","last_undegraded":"2024-03-26T21:22:09.378097+0000","last_fullsized":"2024-03-26T21:22:09.378097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:03:05.033927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.495053+0000","last_change":"2024-03-26T21:21:29.197343+0000","last_active":"2024-03-26T21:22:09.495053+0000","last_peered":"2024-03-26T21:22:09.495053+0000","last_clean":"2024-03-26T21:22:09.495053+0000","last_became_active":"2024-03-26T21:21:29.197027+0000","last_became_peered":"2024-03-26T21:21:29.197027+0000","last_unstale":"2024-03-26T21:22:09.495053+0000","last_undegraded":"2024-03-26T21:22:09.495053+0000","last_fullsized":"2024-03-26T21:22:09.495053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:53:35.883369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374677+0000","last_change":"2024-03-26T21:21:25.187709+0000","last_active":"2024-03-26T21:22:09.374677+0000","last_peered":"2024-03-26T21:22:09.374677+0000","last_clean":"2024-03-26T21:22:09.374677+0000","last_became_active":"2024-03-26T21:21:25.187425+0000","last_became_peered":"2024-03-26T21:21:25.187425+0000","last_unstale":"2024-03-26T21:22:09.374677+0000","last_undegraded":"2024-03-26T21:22:09.374677+0000","last_fullsized":"2024-03-26T21:22:09.374677+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:50:23.189954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840644+0000","last_change":"2024-03-26T21:21:27.198028+0000","last_active":"2024-03-26T21:22:09.840644+0000","last_peered":"2024-03-26T21:22:09.840644+0000","last_clean":"2024-03-26T21:22:09.840644+0000","last_became_active":"2024-03-26T21:21:27.197726+0000","last_became_peered":"2024-03-26T21:21:27.197726+0000","last_unstale":"2024-03-26T21:22:09.840644+0000","last_undegraded":"2024-03-26T21:22:09.840644+0000","last_fullsized":"2024-03-26T21:22:09.840644+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:07:21.862157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840163+0000","last_change":"2024-03-26T21:21:31.213514+0000","last_active":"2024-03-26T21:22:09.840163+0000","last_peered":"2024-03-26T21:22:09.840163+0000","last_clean":"2024-03-26T21:22:09.840163+0000","last_became_active":"2024-03-26T21:21:31.212790+0000","last_became_peered":"2024-03-26T21:21:31.212790+0000","last_unstale":"2024-03-26T21:22:09.840163+0000","last_undegraded":"2024-03-26T21:22:09.840163+0000","last_fullsized":"2024-03-26T21:22:09.840163+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:59:43.621549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248842+0000","last_change":"2024-03-26T21:21:29.198690+0000","last_active":"2024-03-26T21:22:40.248842+0000","last_peered":"2024-03-26T21:22:40.248842+0000","last_clean":"2024-03-26T21:22:40.248842+0000","last_became_active":"2024-03-26T21:21:29.198351+0000","last_became_peered":"2024-03-26T21:21:29.198351+0000","last_unstale":"2024-03-26T21:22:40.248842+0000","last_undegraded":"2024-03-26T21:22:40.248842+0000","last_fullsized":"2024-03-26T21:22:40.248842+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:33:43.550517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838403+0000","last_change":"2024-03-26T21:21:25.196429+0000","last_active":"2024-03-26T21:22:09.838403+0000","last_peered":"2024-03-26T21:22:09.838403+0000","last_clean":"2024-03-26T21:22:09.838403+0000","last_became_active":"2024-03-26T21:21:25.195391+0000","last_became_peered":"2024-03-26T21:21:25.195391+0000","last_unstale":"2024-03-26T21:22:09.838403+0000","last_undegraded":"2024-03-26T21:22:09.838403+0000","last_fullsized":"2024-03-26T21:22:09.838403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:36:31.294183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494920+0000","last_change":"2024-03-26T21:21:27.200636+0000","last_active":"2024-03-26T21:22:09.494920+0000","last_peered":"2024-03-26T21:22:09.494920+0000","last_clean":"2024-03-26T21:22:09.494920+0000","last_became_active":"2024-03-26T21:21:27.200174+0000","last_became_peered":"2024-03-26T21:21:27.200174+0000","last_unstale":"2024-03-26T21:22:09.494920+0000","last_undegraded":"2024-03-26T21:22:09.494920+0000","last_fullsized":"2024-03-26T21:22:09.494920+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:18:28.643508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375483+0000","last_change":"2024-03-26T21:21:31.347515+0000","last_active":"2024-03-26T21:22:09.375483+0000","last_peered":"2024-03-26T21:22:09.375483+0000","last_clean":"2024-03-26T21:22:09.375483+0000","last_became_active":"2024-03-26T21:21:31.347207+0000","last_became_peered":"2024-03-26T21:21:31.347207+0000","last_unstale":"2024-03-26T21:22:09.375483+0000","last_undegraded":"2024-03-26T21:22:09.375483+0000","last_fullsized":"2024-03-26T21:22:09.375483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:17:27.787975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839583+0000","last_change":"2024-03-26T21:21:29.202118+0000","last_active":"2024-03-26T21:22:09.839583+0000","last_peered":"2024-03-26T21:22:09.839583+0000","last_clean":"2024-03-26T21:22:09.839583+0000","last_became_active":"2024-03-26T21:21:29.201813+0000","last_became_peered":"2024-03-26T21:21:29.201813+0000","last_unstale":"2024-03-26T21:22:09.839583+0000","last_undegraded":"2024-03-26T21:22:09.839583+0000","last_fullsized":"2024-03-26T21:22:09.839583+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:24:21.520611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840885+0000","last_change":"2024-03-26T21:21:25.194344+0000","last_active":"2024-03-26T21:22:09.840885+0000","last_peered":"2024-03-26T21:22:09.840885+0000","last_clean":"2024-03-26T21:22:09.840885+0000","last_became_active":"2024-03-26T21:21:25.194061+0000","last_became_peered":"2024-03-26T21:21:25.194061+0000","last_unstale":"2024-03-26T21:22:09.840885+0000","last_undegraded":"2024-03-26T21:22:09.840885+0000","last_fullsized":"2024-03-26T21:22:09.840885+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:34:47.923066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374962+0000","last_change":"2024-03-26T21:21:27.205170+0000","last_active":"2024-03-26T21:22:09.374962+0000","last_peered":"2024-03-26T21:22:09.374962+0000","last_clean":"2024-03-26T21:22:09.374962+0000","last_became_active":"2024-03-26T21:21:27.203150+0000","last_became_peered":"2024-03-26T21:21:27.203150+0000","last_unstale":"2024-03-26T21:22:09.374962+0000","last_undegraded":"2024-03-26T21:22:09.374962+0000","last_fullsized":"2024-03-26T21:22:09.374962+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:49:06.759997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839396+0000","last_change":"2024-03-26T21:21:31.212695+0000","last_active":"2024-03-26T21:22:09.839396+0000","last_peered":"2024-03-26T21:22:09.839396+0000","last_clean":"2024-03-26T21:22:09.839396+0000","last_became_active":"2024-03-26T21:21:31.212361+0000","last_became_peered":"2024-03-26T21:21:31.212361+0000","last_unstale":"2024-03-26T21:22:09.839396+0000","last_undegraded":"2024-03-26T21:22:09.839396+0000","last_fullsized":"2024-03-26T21:22:09.839396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:43:43.165565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840565+0000","last_change":"2024-03-26T21:21:29.200516+0000","last_active":"2024-03-26T21:22:09.840565+0000","last_peered":"2024-03-26T21:22:09.840565+0000","last_clean":"2024-03-26T21:22:09.840565+0000","last_became_active":"2024-03-26T21:21:29.200203+0000","last_became_peered":"2024-03-26T21:21:29.200203+0000","last_unstale":"2024-03-26T21:22:09.840565+0000","last_undegraded":"2024-03-26T21:22:09.840565+0000","last_fullsized":"2024-03-26T21:22:09.840565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:57:23.985115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839817+0000","last_change":"2024-03-26T21:21:25.194146+0000","last_active":"2024-03-26T21:22:09.839817+0000","last_peered":"2024-03-26T21:22:09.839817+0000","last_clean":"2024-03-26T21:22:09.839817+0000","last_became_active":"2024-03-26T21:21:25.193199+0000","last_became_peered":"2024-03-26T21:21:25.193199+0000","last_unstale":"2024-03-26T21:22:09.839817+0000","last_undegraded":"2024-03-26T21:22:09.839817+0000","last_fullsized":"2024-03-26T21:22:09.839817+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:50:20.411801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374611+0000","last_change":"2024-03-26T21:21:27.203002+0000","last_active":"2024-03-26T21:22:09.374611+0000","last_peered":"2024-03-26T21:22:09.374611+0000","last_clean":"2024-03-26T21:22:09.374611+0000","last_became_active":"2024-03-26T21:21:27.202176+0000","last_became_peered":"2024-03-26T21:21:27.202176+0000","last_unstale":"2024-03-26T21:22:09.374611+0000","last_undegraded":"2024-03-26T21:22:09.374611+0000","last_fullsized":"2024-03-26T21:22:09.374611+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:31:54.665171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374467+0000","last_change":"2024-03-26T21:21:31.216443+0000","last_active":"2024-03-26T21:22:09.374467+0000","last_peered":"2024-03-26T21:22:09.374467+0000","last_clean":"2024-03-26T21:22:09.374467+0000","last_became_active":"2024-03-26T21:21:31.215690+0000","last_became_peered":"2024-03-26T21:21:31.215690+0000","last_unstale":"2024-03-26T21:22:09.374467+0000","last_undegraded":"2024-03-26T21:22:09.374467+0000","last_fullsized":"2024-03-26T21:22:09.374467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:08:12.234978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377145+0000","last_change":"2024-03-26T21:21:29.206135+0000","last_active":"2024-03-26T21:22:09.377145+0000","last_peered":"2024-03-26T21:22:09.377145+0000","last_clean":"2024-03-26T21:22:09.377145+0000","last_became_active":"2024-03-26T21:21:29.205689+0000","last_became_peered":"2024-03-26T21:21:29.205689+0000","last_unstale":"2024-03-26T21:22:09.377145+0000","last_undegraded":"2024-03-26T21:22:09.377145+0000","last_fullsized":"2024-03-26T21:22:09.377145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:54:31.565734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378234+0000","last_change":"2024-03-26T21:21:25.193452+0000","last_active":"2024-03-26T21:22:09.378234+0000","last_peered":"2024-03-26T21:22:09.378234+0000","last_clean":"2024-03-26T21:22:09.378234+0000","last_became_active":"2024-03-26T21:21:25.193185+0000","last_became_peered":"2024-03-26T21:21:25.193185+0000","last_unstale":"2024-03-26T21:22:09.378234+0000","last_undegraded":"2024-03-26T21:22:09.378234+0000","last_fullsized":"2024-03-26T21:22:09.378234+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:34:43.864243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375855+0000","last_change":"2024-03-26T21:21:27.207383+0000","last_active":"2024-03-26T21:22:09.375855+0000","last_peered":"2024-03-26T21:22:09.375855+0000","last_clean":"2024-03-26T21:22:09.375855+0000","last_became_active":"2024-03-26T21:21:27.206804+0000","last_became_peered":"2024-03-26T21:21:27.206804+0000","last_unstale":"2024-03-26T21:22:09.375855+0000","last_undegraded":"2024-03-26T21:22:09.375855+0000","last_fullsized":"2024-03-26T21:22:09.375855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:18:08.358275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.493866+0000","last_change":"2024-03-26T21:21:31.217993+0000","last_active":"2024-03-26T21:22:09.493866+0000","last_peered":"2024-03-26T21:22:09.493866+0000","last_clean":"2024-03-26T21:22:09.493866+0000","last_became_active":"2024-03-26T21:21:31.217634+0000","last_became_peered":"2024-03-26T21:21:31.217634+0000","last_unstale":"2024-03-26T21:22:09.493866+0000","last_undegraded":"2024-03-26T21:22:09.493866+0000","last_fullsized":"2024-03-26T21:22:09.493866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:01:17.247200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375317+0000","last_change":"2024-03-26T21:21:29.205723+0000","last_active":"2024-03-26T21:22:09.375317+0000","last_peered":"2024-03-26T21:22:09.375317+0000","last_clean":"2024-03-26T21:22:09.375317+0000","last_became_active":"2024-03-26T21:21:29.205369+0000","last_became_peered":"2024-03-26T21:21:29.205369+0000","last_unstale":"2024-03-26T21:22:09.375317+0000","last_undegraded":"2024-03-26T21:22:09.375317+0000","last_fullsized":"2024-03-26T21:22:09.375317+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:55:36.680447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839773+0000","last_change":"2024-03-26T21:21:25.196056+0000","last_active":"2024-03-26T21:22:09.839773+0000","last_peered":"2024-03-26T21:22:09.839773+0000","last_clean":"2024-03-26T21:22:09.839773+0000","last_became_active":"2024-03-26T21:21:25.195560+0000","last_became_peered":"2024-03-26T21:21:25.195560+0000","last_unstale":"2024-03-26T21:22:09.839773+0000","last_undegraded":"2024-03-26T21:22:09.839773+0000","last_fullsized":"2024-03-26T21:22:09.839773+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:26:35.588884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838676+0000","last_change":"2024-03-26T21:21:48.182304+0000","last_active":"2024-03-26T21:22:09.838676+0000","last_peered":"2024-03-26T21:22:09.838676+0000","last_clean":"2024-03-26T21:22:09.838676+0000","last_became_active":"2024-03-26T21:21:48.174368+0000","last_became_peered":"2024-03-26T21:21:48.174368+0000","last_unstale":"2024-03-26T21:22:09.838676+0000","last_undegraded":"2024-03-26T21:22:09.838676+0000","last_fullsized":"2024-03-26T21:22:09.838676+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:39:58.801370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376591+0000","last_change":"2024-03-26T21:21:31.207576+0000","last_active":"2024-03-26T21:22:09.376591+0000","last_peered":"2024-03-26T21:22:09.376591+0000","last_clean":"2024-03-26T21:22:09.376591+0000","last_became_active":"2024-03-26T21:21:31.207258+0000","last_became_peered":"2024-03-26T21:21:31.207258+0000","last_unstale":"2024-03-26T21:22:09.376591+0000","last_undegraded":"2024-03-26T21:22:09.376591+0000","last_fullsized":"2024-03-26T21:22:09.376591+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:44:02.818399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248546+0000","last_change":"2024-03-26T21:21:29.198134+0000","last_active":"2024-03-26T21:22:40.248546+0000","last_peered":"2024-03-26T21:22:40.248546+0000","last_clean":"2024-03-26T21:22:40.248546+0000","last_became_active":"2024-03-26T21:21:29.197160+0000","last_became_peered":"2024-03-26T21:21:29.197160+0000","last_unstale":"2024-03-26T21:22:40.248546+0000","last_undegraded":"2024-03-26T21:22:40.248546+0000","last_fullsized":"2024-03-26T21:22:40.248546+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:29:45.344203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494887+0000","last_change":"2024-03-26T21:21:25.190921+0000","last_active":"2024-03-26T21:22:09.494887+0000","last_peered":"2024-03-26T21:22:09.494887+0000","last_clean":"2024-03-26T21:22:09.494887+0000","last_became_active":"2024-03-26T21:21:25.190608+0000","last_became_peered":"2024-03-26T21:21:25.190608+0000","last_unstale":"2024-03-26T21:22:09.494887+0000","last_undegraded":"2024-03-26T21:22:09.494887+0000","last_fullsized":"2024-03-26T21:22:09.494887+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:37:24.149510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839601+0000","last_change":"2024-03-26T21:21:27.199318+0000","last_active":"2024-03-26T21:22:09.839601+0000","last_peered":"2024-03-26T21:22:09.839601+0000","last_clean":"2024-03-26T21:22:09.839601+0000","last_became_active":"2024-03-26T21:21:27.198803+0000","last_became_peered":"2024-03-26T21:21:27.198803+0000","last_unstale":"2024-03-26T21:22:09.839601+0000","last_undegraded":"2024-03-26T21:22:09.839601+0000","last_fullsized":"2024-03-26T21:22:09.839601+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:51:27.906494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.495061+0000","last_change":"2024-03-26T21:21:31.210765+0000","last_active":"2024-03-26T21:22:09.495061+0000","last_peered":"2024-03-26T21:22:09.495061+0000","last_clean":"2024-03-26T21:22:09.495061+0000","last_became_active":"2024-03-26T21:21:31.210414+0000","last_became_peered":"2024-03-26T21:21:31.210414+0000","last_unstale":"2024-03-26T21:22:09.495061+0000","last_undegraded":"2024-03-26T21:22:09.495061+0000","last_fullsized":"2024-03-26T21:22:09.495061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:08:13.712367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248823+0000","last_change":"2024-03-26T21:21:29.204650+0000","last_active":"2024-03-26T21:22:40.248823+0000","last_peered":"2024-03-26T21:22:40.248823+0000","last_clean":"2024-03-26T21:22:40.248823+0000","last_became_active":"2024-03-26T21:21:29.204354+0000","last_became_peered":"2024-03-26T21:21:29.204354+0000","last_unstale":"2024-03-26T21:22:40.248823+0000","last_undegraded":"2024-03-26T21:22:40.248823+0000","last_fullsized":"2024-03-26T21:22:40.248823+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:49:36.163742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838959+0000","last_change":"2024-03-26T21:21:25.194063+0000","last_active":"2024-03-26T21:22:09.838959+0000","last_peered":"2024-03-26T21:22:09.838959+0000","last_clean":"2024-03-26T21:22:09.838959+0000","last_became_active":"2024-03-26T21:21:25.193176+0000","last_became_peered":"2024-03-26T21:21:25.193176+0000","last_unstale":"2024-03-26T21:22:09.838959+0000","last_undegraded":"2024-03-26T21:22:09.838959+0000","last_fullsized":"2024-03-26T21:22:09.838959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:56:46.803191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376643+0000","last_change":"2024-03-26T21:21:27.204286+0000","last_active":"2024-03-26T21:22:09.376643+0000","last_peered":"2024-03-26T21:22:09.376643+0000","last_clean":"2024-03-26T21:22:09.376643+0000","last_became_active":"2024-03-26T21:21:27.203710+0000","last_became_peered":"2024-03-26T21:21:27.203710+0000","last_unstale":"2024-03-26T21:22:09.376643+0000","last_undegraded":"2024-03-26T21:22:09.376643+0000","last_fullsized":"2024-03-26T21:22:09.376643+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:11:57.880443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840023+0000","last_change":"2024-03-26T21:21:31.348447+0000","last_active":"2024-03-26T21:22:09.840023+0000","last_peered":"2024-03-26T21:22:09.840023+0000","last_clean":"2024-03-26T21:22:09.840023+0000","last_became_active":"2024-03-26T21:21:31.348120+0000","last_became_peered":"2024-03-26T21:21:31.348120+0000","last_unstale":"2024-03-26T21:22:09.840023+0000","last_undegraded":"2024-03-26T21:22:09.840023+0000","last_fullsized":"2024-03-26T21:22:09.840023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:46:17.480158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839614+0000","last_change":"2024-03-26T21:21:29.205319+0000","last_active":"2024-03-26T21:22:09.839614+0000","last_peered":"2024-03-26T21:22:09.839614+0000","last_clean":"2024-03-26T21:22:09.839614+0000","last_became_active":"2024-03-26T21:21:29.205024+0000","last_became_peered":"2024-03-26T21:21:29.205024+0000","last_unstale":"2024-03-26T21:22:09.839614+0000","last_undegraded":"2024-03-26T21:22:09.839614+0000","last_fullsized":"2024-03-26T21:22:09.839614+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:07:59.496867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840638+0000","last_change":"2024-03-26T21:21:25.191540+0000","last_active":"2024-03-26T21:22:09.840638+0000","last_peered":"2024-03-26T21:22:09.840638+0000","last_clean":"2024-03-26T21:22:09.840638+0000","last_became_active":"2024-03-26T21:21:25.191291+0000","last_became_peered":"2024-03-26T21:21:25.191291+0000","last_unstale":"2024-03-26T21:22:09.840638+0000","last_undegraded":"2024-03-26T21:22:09.840638+0000","last_fullsized":"2024-03-26T21:22:09.840638+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:00:44.268906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376049+0000","last_change":"2024-03-26T21:21:27.204276+0000","last_active":"2024-03-26T21:22:09.376049+0000","last_peered":"2024-03-26T21:22:09.376049+0000","last_clean":"2024-03-26T21:22:09.376049+0000","last_became_active":"2024-03-26T21:21:27.203590+0000","last_became_peered":"2024-03-26T21:21:27.203590+0000","last_unstale":"2024-03-26T21:22:09.376049+0000","last_undegraded":"2024-03-26T21:22:09.376049+0000","last_fullsized":"2024-03-26T21:22:09.376049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:03:32.003907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838418+0000","last_change":"2024-03-26T21:21:31.212118+0000","last_active":"2024-03-26T21:22:09.838418+0000","last_peered":"2024-03-26T21:22:09.838418+0000","last_clean":"2024-03-26T21:22:09.838418+0000","last_became_active":"2024-03-26T21:21:31.211722+0000","last_became_peered":"2024-03-26T21:21:31.211722+0000","last_unstale":"2024-03-26T21:22:09.838418+0000","last_undegraded":"2024-03-26T21:22:09.838418+0000","last_fullsized":"2024-03-26T21:22:09.838418+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:36:10.162393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376254+0000","last_change":"2024-03-26T21:21:29.206711+0000","last_active":"2024-03-26T21:22:09.376254+0000","last_peered":"2024-03-26T21:22:09.376254+0000","last_clean":"2024-03-26T21:22:09.376254+0000","last_became_active":"2024-03-26T21:21:29.206404+0000","last_became_peered":"2024-03-26T21:21:29.206404+0000","last_unstale":"2024-03-26T21:22:09.376254+0000","last_undegraded":"2024-03-26T21:22:09.376254+0000","last_fullsized":"2024-03-26T21:22:09.376254+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:31:03.673863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838193+0000","last_change":"2024-03-26T21:21:25.189724+0000","last_active":"2024-03-26T21:22:09.838193+0000","last_peered":"2024-03-26T21:22:09.838193+0000","last_clean":"2024-03-26T21:22:09.838193+0000","last_became_active":"2024-03-26T21:21:25.189454+0000","last_became_peered":"2024-03-26T21:21:25.189454+0000","last_unstale":"2024-03-26T21:22:09.838193+0000","last_undegraded":"2024-03-26T21:22:09.838193+0000","last_fullsized":"2024-03-26T21:22:09.838193+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:09:42.669740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494684+0000","last_change":"2024-03-26T21:21:27.199475+0000","last_active":"2024-03-26T21:22:09.494684+0000","last_peered":"2024-03-26T21:22:09.494684+0000","last_clean":"2024-03-26T21:22:09.494684+0000","last_became_active":"2024-03-26T21:21:27.199144+0000","last_became_peered":"2024-03-26T21:21:27.199144+0000","last_unstale":"2024-03-26T21:22:09.494684+0000","last_undegraded":"2024-03-26T21:22:09.494684+0000","last_fullsized":"2024-03-26T21:22:09.494684+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:46:09.380308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375399+0000","last_change":"2024-03-26T21:21:31.219944+0000","last_active":"2024-03-26T21:22:09.375399+0000","last_peered":"2024-03-26T21:22:09.375399+0000","last_clean":"2024-03-26T21:22:09.375399+0000","last_became_active":"2024-03-26T21:21:31.219577+0000","last_became_peered":"2024-03-26T21:21:31.219577+0000","last_unstale":"2024-03-26T21:22:09.375399+0000","last_undegraded":"2024-03-26T21:22:09.375399+0000","last_fullsized":"2024-03-26T21:22:09.375399+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:22:57.749637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376315+0000","last_change":"2024-03-26T21:21:25.188671+0000","last_active":"2024-03-26T21:22:09.376315+0000","last_peered":"2024-03-26T21:22:09.376315+0000","last_clean":"2024-03-26T21:22:09.376315+0000","last_became_active":"2024-03-26T21:21:25.187132+0000","last_became_peered":"2024-03-26T21:21:25.187132+0000","last_unstale":"2024-03-26T21:22:09.376315+0000","last_undegraded":"2024-03-26T21:22:09.376315+0000","last_fullsized":"2024-03-26T21:22:09.376315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:07:55.551378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375609+0000","last_change":"2024-03-26T21:21:27.208795+0000","last_active":"2024-03-26T21:22:09.375609+0000","last_peered":"2024-03-26T21:22:09.375609+0000","last_clean":"2024-03-26T21:22:09.375609+0000","last_became_active":"2024-03-26T21:21:27.208450+0000","last_became_peered":"2024-03-26T21:21:27.208450+0000","last_unstale":"2024-03-26T21:22:09.375609+0000","last_undegraded":"2024-03-26T21:22:09.375609+0000","last_fullsized":"2024-03-26T21:22:09.375609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:24:43.770586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838190+0000","last_change":"2024-03-26T21:21:29.209719+0000","last_active":"2024-03-26T21:22:09.838190+0000","last_peered":"2024-03-26T21:22:09.838190+0000","last_clean":"2024-03-26T21:22:09.838190+0000","last_became_active":"2024-03-26T21:21:29.209351+0000","last_became_peered":"2024-03-26T21:21:29.209351+0000","last_unstale":"2024-03-26T21:22:09.838190+0000","last_undegraded":"2024-03-26T21:22:09.838190+0000","last_fullsized":"2024-03-26T21:22:09.838190+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:30:58.632440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.493883+0000","last_change":"2024-03-26T21:21:31.210011+0000","last_active":"2024-03-26T21:22:09.493883+0000","last_peered":"2024-03-26T21:22:09.493883+0000","last_clean":"2024-03-26T21:22:09.493883+0000","last_became_active":"2024-03-26T21:21:31.209693+0000","last_became_peered":"2024-03-26T21:21:31.209693+0000","last_unstale":"2024-03-26T21:22:09.493883+0000","last_undegraded":"2024-03-26T21:22:09.493883+0000","last_fullsized":"2024-03-26T21:22:09.493883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:52:13.737074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377796+0000","last_change":"2024-03-26T21:21:25.188069+0000","last_active":"2024-03-26T21:22:09.377796+0000","last_peered":"2024-03-26T21:22:09.377796+0000","last_clean":"2024-03-26T21:22:09.377796+0000","last_became_active":"2024-03-26T21:21:25.187545+0000","last_became_peered":"2024-03-26T21:21:25.187545+0000","last_unstale":"2024-03-26T21:22:09.377796+0000","last_undegraded":"2024-03-26T21:22:09.377796+0000","last_fullsized":"2024-03-26T21:22:09.377796+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:45:58.086524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377019+0000","last_change":"2024-03-26T21:21:27.204561+0000","last_active":"2024-03-26T21:22:09.377019+0000","last_peered":"2024-03-26T21:22:09.377019+0000","last_clean":"2024-03-26T21:22:09.377019+0000","last_became_active":"2024-03-26T21:21:27.203458+0000","last_became_peered":"2024-03-26T21:21:27.203458+0000","last_unstale":"2024-03-26T21:22:09.377019+0000","last_undegraded":"2024-03-26T21:22:09.377019+0000","last_fullsized":"2024-03-26T21:22:09.377019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:45:59.929399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248379+0000","last_change":"2024-03-26T21:21:29.199024+0000","last_active":"2024-03-26T21:22:40.248379+0000","last_peered":"2024-03-26T21:22:40.248379+0000","last_clean":"2024-03-26T21:22:40.248379+0000","last_became_active":"2024-03-26T21:21:29.198694+0000","last_became_peered":"2024-03-26T21:21:29.198694+0000","last_unstale":"2024-03-26T21:22:40.248379+0000","last_undegraded":"2024-03-26T21:22:40.248379+0000","last_fullsized":"2024-03-26T21:22:40.248379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:44:20.562098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375771+0000","last_change":"2024-03-26T21:21:31.213146+0000","last_active":"2024-03-26T21:22:09.375771+0000","last_peered":"2024-03-26T21:22:09.375771+0000","last_clean":"2024-03-26T21:22:09.375771+0000","last_became_active":"2024-03-26T21:21:31.212791+0000","last_became_peered":"2024-03-26T21:21:31.212791+0000","last_unstale":"2024-03-26T21:22:09.375771+0000","last_undegraded":"2024-03-26T21:22:09.375771+0000","last_fullsized":"2024-03-26T21:22:09.375771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:10:23.445388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839060+0000","last_change":"2024-03-26T21:21:25.195053+0000","last_active":"2024-03-26T21:22:09.839060+0000","last_peered":"2024-03-26T21:22:09.839060+0000","last_clean":"2024-03-26T21:22:09.839060+0000","last_became_active":"2024-03-26T21:21:25.194460+0000","last_became_peered":"2024-03-26T21:21:25.194460+0000","last_unstale":"2024-03-26T21:22:09.839060+0000","last_undegraded":"2024-03-26T21:22:09.839060+0000","last_fullsized":"2024-03-26T21:22:09.839060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:31:47.477889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":94,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248400+0000","last_change":"2024-03-26T21:21:27.199020+0000","last_active":"2024-03-26T21:22:40.248400+0000","last_peered":"2024-03-26T21:22:40.248400+0000","last_clean":"2024-03-26T21:22:40.248400+0000","last_became_active":"2024-03-26T21:21:27.198722+0000","last_became_peered":"2024-03-26T21:21:27.198722+0000","last_unstale":"2024-03-26T21:22:40.248400+0000","last_undegraded":"2024-03-26T21:22:40.248400+0000","last_fullsized":"2024-03-26T21:22:40.248400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:54:48.210382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494822+0000","last_change":"2024-03-26T21:21:29.205845+0000","last_active":"2024-03-26T21:22:09.494822+0000","last_peered":"2024-03-26T21:22:09.494822+0000","last_clean":"2024-03-26T21:22:09.494822+0000","last_became_active":"2024-03-26T21:21:29.205332+0000","last_became_peered":"2024-03-26T21:21:29.205332+0000","last_unstale":"2024-03-26T21:22:09.494822+0000","last_undegraded":"2024-03-26T21:22:09.494822+0000","last_fullsized":"2024-03-26T21:22:09.494822+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:32:49.582713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840457+0000","last_change":"2024-03-26T21:21:31.349116+0000","last_active":"2024-03-26T21:22:09.840457+0000","last_peered":"2024-03-26T21:22:09.840457+0000","last_clean":"2024-03-26T21:22:09.840457+0000","last_became_active":"2024-03-26T21:21:31.348805+0000","last_became_peered":"2024-03-26T21:21:31.348805+0000","last_unstale":"2024-03-26T21:22:09.840457+0000","last_undegraded":"2024-03-26T21:22:09.840457+0000","last_fullsized":"2024-03-26T21:22:09.840457+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:30:25.112916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'5","reported_seq":117,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:44.060668+0000","last_change":"2024-03-26T21:21:29.205538+0000","last_active":"2024-03-26T21:22:44.060668+0000","last_peered":"2024-03-26T21:22:44.060668+0000","last_clean":"2024-03-26T21:22:44.060668+0000","last_became_active":"2024-03-26T21:21:25.188351+0000","last_became_peered":"2024-03-26T21:21:25.188351+0000","last_unstale":"2024-03-26T21:22:44.060668+0000","last_undegraded":"2024-03-26T21:22:44.060668+0000","last_fullsized":"2024-03-26T21:22:44.060668+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:21:34.467438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00100063,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838161+0000","last_change":"2024-03-26T21:21:25.184917+0000","last_active":"2024-03-26T21:22:09.838161+0000","last_peered":"2024-03-26T21:22:09.838161+0000","last_clean":"2024-03-26T21:22:09.838161+0000","last_became_active":"2024-03-26T21:21:25.184619+0000","last_became_peered":"2024-03-26T21:21:25.184619+0000","last_unstale":"2024-03-26T21:22:09.838161+0000","last_undegraded":"2024-03-26T21:22:09.838161+0000","last_fullsized":"2024-03-26T21:22:09.838161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:26:03.908836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.379149+0000","last_change":"2024-03-26T21:21:27.195400+0000","last_active":"2024-03-26T21:22:09.379149+0000","last_peered":"2024-03-26T21:22:09.379149+0000","last_clean":"2024-03-26T21:22:09.379149+0000","last_became_active":"2024-03-26T21:21:27.195082+0000","last_became_peered":"2024-03-26T21:21:27.195082+0000","last_unstale":"2024-03-26T21:22:09.379149+0000","last_undegraded":"2024-03-26T21:22:09.379149+0000","last_fullsized":"2024-03-26T21:22:09.379149+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:40:57.955259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838613+0000","last_change":"2024-03-26T21:21:29.204082+0000","last_active":"2024-03-26T21:22:09.838613+0000","last_peered":"2024-03-26T21:22:09.838613+0000","last_clean":"2024-03-26T21:22:09.838613+0000","last_became_active":"2024-03-26T21:21:29.203741+0000","last_became_peered":"2024-03-26T21:21:29.203741+0000","last_unstale":"2024-03-26T21:22:09.838613+0000","last_undegraded":"2024-03-26T21:22:09.838613+0000","last_fullsized":"2024-03-26T21:22:09.838613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:15:20.663676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840378+0000","last_change":"2024-03-26T21:21:31.214988+0000","last_active":"2024-03-26T21:22:09.840378+0000","last_peered":"2024-03-26T21:22:09.840378+0000","last_clean":"2024-03-26T21:22:09.840378+0000","last_became_active":"2024-03-26T21:21:31.214654+0000","last_became_peered":"2024-03-26T21:21:31.214654+0000","last_unstale":"2024-03-26T21:22:09.840378+0000","last_undegraded":"2024-03-26T21:22:09.840378+0000","last_fullsized":"2024-03-26T21:22:09.840378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:48:12.886918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375028+0000","last_change":"2024-03-26T21:21:29.194135+0000","last_active":"2024-03-26T21:22:09.375028+0000","last_peered":"2024-03-26T21:22:09.375028+0000","last_clean":"2024-03-26T21:22:09.375028+0000","last_became_active":"2024-03-26T21:21:25.193663+0000","last_became_peered":"2024-03-26T21:21:25.193663+0000","last_unstale":"2024-03-26T21:22:09.375028+0000","last_undegraded":"2024-03-26T21:22:09.375028+0000","last_fullsized":"2024-03-26T21:22:09.375028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:40:57.855172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002579769,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838769+0000","last_change":"2024-03-26T21:21:48.177755+0000","last_active":"2024-03-26T21:22:09.838769+0000","last_peered":"2024-03-26T21:22:09.838769+0000","last_clean":"2024-03-26T21:22:09.838769+0000","last_became_active":"2024-03-26T21:21:48.177292+0000","last_became_peered":"2024-03-26T21:21:48.177292+0000","last_unstale":"2024-03-26T21:22:09.838769+0000","last_undegraded":"2024-03-26T21:22:09.838769+0000","last_fullsized":"2024-03-26T21:22:09.838769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:08:22.638059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'128","reported_seq":272,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:11.473088+0000","last_change":"2024-03-26T21:21:14.714857+0000","last_active":"2024-03-26T21:22:11.473088+0000","last_peered":"2024-03-26T21:22:11.473088+0000","last_clean":"2024-03-26T21:22:11.473088+0000","last_became_active":"2024-03-26T21:21:14.714386+0000","last_became_peered":"2024-03-26T21:21:14.714386+0000","last_unstale":"2024-03-26T21:22:11.473088+0000","last_undegraded":"2024-03-26T21:22:11.473088+0000","last_fullsized":"2024-03-26T21:22:11.473088+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:19:03.226555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:19:03.226555+0000","last_clean_scrub_stamp":"2024-03-26T21:19:03.226555+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:23:38.758836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248855+0000","last_change":"2024-03-26T21:21:27.197740+0000","last_active":"2024-03-26T21:22:40.248855+0000","last_peered":"2024-03-26T21:22:40.248855+0000","last_clean":"2024-03-26T21:22:40.248855+0000","last_became_active":"2024-03-26T21:21:27.197389+0000","last_became_peered":"2024-03-26T21:21:27.197389+0000","last_unstale":"2024-03-26T21:22:40.248855+0000","last_undegraded":"2024-03-26T21:22:40.248855+0000","last_fullsized":"2024-03-26T21:22:40.248855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:20:02.610847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840003+0000","last_change":"2024-03-26T21:21:29.199860+0000","last_active":"2024-03-26T21:22:09.840003+0000","last_peered":"2024-03-26T21:22:09.840003+0000","last_clean":"2024-03-26T21:22:09.840003+0000","last_became_active":"2024-03-26T21:21:29.199549+0000","last_became_peered":"2024-03-26T21:21:29.199549+0000","last_unstale":"2024-03-26T21:22:09.840003+0000","last_undegraded":"2024-03-26T21:22:09.840003+0000","last_fullsized":"2024-03-26T21:22:09.840003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:13:43.893929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377566+0000","last_change":"2024-03-26T21:21:31.218936+0000","last_active":"2024-03-26T21:22:09.377566+0000","last_peered":"2024-03-26T21:22:09.377566+0000","last_clean":"2024-03-26T21:22:09.377566+0000","last_became_active":"2024-03-26T21:21:31.218620+0000","last_became_peered":"2024-03-26T21:21:31.218620+0000","last_unstale":"2024-03-26T21:22:09.377566+0000","last_undegraded":"2024-03-26T21:22:09.377566+0000","last_fullsized":"2024-03-26T21:22:09.377566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:27:44.780741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494329+0000","last_change":"2024-03-26T21:21:29.195555+0000","last_active":"2024-03-26T21:22:09.494329+0000","last_peered":"2024-03-26T21:22:09.494329+0000","last_clean":"2024-03-26T21:22:09.494329+0000","last_became_active":"2024-03-26T21:21:25.192273+0000","last_became_peered":"2024-03-26T21:21:25.192273+0000","last_unstale":"2024-03-26T21:22:09.494329+0000","last_undegraded":"2024-03-26T21:22:09.494329+0000","last_fullsized":"2024-03-26T21:22:09.494329+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:59:22.247736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039106999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839177+0000","last_change":"2024-03-26T21:21:25.195379+0000","last_active":"2024-03-26T21:22:09.839177+0000","last_peered":"2024-03-26T21:22:09.839177+0000","last_clean":"2024-03-26T21:22:09.839177+0000","last_became_active":"2024-03-26T21:21:25.195119+0000","last_became_peered":"2024-03-26T21:21:25.195119+0000","last_unstale":"2024-03-26T21:22:09.839177+0000","last_undegraded":"2024-03-26T21:22:09.839177+0000","last_fullsized":"2024-03-26T21:22:09.839177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:44:22.259071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494545+0000","last_change":"2024-03-26T21:21:48.188101+0000","last_active":"2024-03-26T21:22:09.494545+0000","last_peered":"2024-03-26T21:22:09.494545+0000","last_clean":"2024-03-26T21:22:09.494545+0000","last_became_active":"2024-03-26T21:21:48.177016+0000","last_became_peered":"2024-03-26T21:21:48.177016+0000","last_unstale":"2024-03-26T21:22:09.494545+0000","last_undegraded":"2024-03-26T21:22:09.494545+0000","last_fullsized":"2024-03-26T21:22:09.494545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:59:47.697063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840289+0000","last_change":"2024-03-26T21:21:29.208167+0000","last_active":"2024-03-26T21:22:09.840289+0000","last_peered":"2024-03-26T21:22:09.840289+0000","last_clean":"2024-03-26T21:22:09.840289+0000","last_became_active":"2024-03-26T21:21:29.207799+0000","last_became_peered":"2024-03-26T21:21:29.207799+0000","last_unstale":"2024-03-26T21:22:09.840289+0000","last_undegraded":"2024-03-26T21:22:09.840289+0000","last_fullsized":"2024-03-26T21:22:09.840289+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:29:56.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376278+0000","last_change":"2024-03-26T21:21:31.348097+0000","last_active":"2024-03-26T21:22:09.376278+0000","last_peered":"2024-03-26T21:22:09.376278+0000","last_clean":"2024-03-26T21:22:09.376278+0000","last_became_active":"2024-03-26T21:21:31.347734+0000","last_became_peered":"2024-03-26T21:21:31.347734+0000","last_unstale":"2024-03-26T21:22:09.376278+0000","last_undegraded":"2024-03-26T21:22:09.376278+0000","last_fullsized":"2024-03-26T21:22:09.376278+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:46:26.345041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840493+0000","last_change":"2024-03-26T21:21:25.190385+0000","last_active":"2024-03-26T21:22:09.840493+0000","last_peered":"2024-03-26T21:22:09.840493+0000","last_clean":"2024-03-26T21:22:09.840493+0000","last_became_active":"2024-03-26T21:21:25.189649+0000","last_became_peered":"2024-03-26T21:21:25.189649+0000","last_unstale":"2024-03-26T21:22:09.840493+0000","last_undegraded":"2024-03-26T21:22:09.840493+0000","last_fullsized":"2024-03-26T21:22:09.840493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:19:57.794225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494757+0000","last_change":"2024-03-26T21:21:27.193764+0000","last_active":"2024-03-26T21:22:09.494757+0000","last_peered":"2024-03-26T21:22:09.494757+0000","last_clean":"2024-03-26T21:22:09.494757+0000","last_became_active":"2024-03-26T21:21:27.193424+0000","last_became_peered":"2024-03-26T21:21:27.193424+0000","last_unstale":"2024-03-26T21:22:09.494757+0000","last_undegraded":"2024-03-26T21:22:09.494757+0000","last_fullsized":"2024-03-26T21:22:09.494757+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:46:44.280198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839267+0000","last_change":"2024-03-26T21:21:29.198259+0000","last_active":"2024-03-26T21:22:09.839267+0000","last_peered":"2024-03-26T21:22:09.839267+0000","last_clean":"2024-03-26T21:22:09.839267+0000","last_became_active":"2024-03-26T21:21:29.197988+0000","last_became_peered":"2024-03-26T21:21:29.197988+0000","last_unstale":"2024-03-26T21:22:09.839267+0000","last_undegraded":"2024-03-26T21:22:09.839267+0000","last_fullsized":"2024-03-26T21:22:09.839267+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:56:53.580647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375976+0000","last_change":"2024-03-26T21:21:31.209675+0000","last_active":"2024-03-26T21:22:09.375976+0000","last_peered":"2024-03-26T21:22:09.375976+0000","last_clean":"2024-03-26T21:22:09.375976+0000","last_became_active":"2024-03-26T21:21:31.209349+0000","last_became_peered":"2024-03-26T21:21:31.209349+0000","last_unstale":"2024-03-26T21:22:09.375976+0000","last_undegraded":"2024-03-26T21:22:09.375976+0000","last_fullsized":"2024-03-26T21:22:09.375976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:02:35.862171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378837+0000","last_change":"2024-03-26T21:21:25.194315+0000","last_active":"2024-03-26T21:22:09.378837+0000","last_peered":"2024-03-26T21:22:09.378837+0000","last_clean":"2024-03-26T21:22:09.378837+0000","last_became_active":"2024-03-26T21:21:25.193875+0000","last_became_peered":"2024-03-26T21:21:25.193875+0000","last_unstale":"2024-03-26T21:22:09.378837+0000","last_undegraded":"2024-03-26T21:22:09.378837+0000","last_fullsized":"2024-03-26T21:22:09.378837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:21:43.442879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374492+0000","last_change":"2024-03-26T21:21:27.204833+0000","last_active":"2024-03-26T21:22:09.374492+0000","last_peered":"2024-03-26T21:22:09.374492+0000","last_clean":"2024-03-26T21:22:09.374492+0000","last_became_active":"2024-03-26T21:21:27.204524+0000","last_became_peered":"2024-03-26T21:21:27.204524+0000","last_unstale":"2024-03-26T21:22:09.374492+0000","last_undegraded":"2024-03-26T21:22:09.374492+0000","last_fullsized":"2024-03-26T21:22:09.374492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:09:56.359121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375749+0000","last_change":"2024-03-26T21:21:29.198837+0000","last_active":"2024-03-26T21:22:09.375749+0000","last_peered":"2024-03-26T21:22:09.375749+0000","last_clean":"2024-03-26T21:22:09.375749+0000","last_became_active":"2024-03-26T21:21:29.198142+0000","last_became_peered":"2024-03-26T21:21:29.198142+0000","last_unstale":"2024-03-26T21:22:09.375749+0000","last_undegraded":"2024-03-26T21:22:09.375749+0000","last_fullsized":"2024-03-26T21:22:09.375749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:41:27.234335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840939+0000","last_change":"2024-03-26T21:21:31.348461+0000","last_active":"2024-03-26T21:22:09.840939+0000","last_peered":"2024-03-26T21:22:09.840939+0000","last_clean":"2024-03-26T21:22:09.840939+0000","last_became_active":"2024-03-26T21:21:31.348127+0000","last_became_peered":"2024-03-26T21:21:31.348127+0000","last_unstale":"2024-03-26T21:22:09.840939+0000","last_undegraded":"2024-03-26T21:22:09.840939+0000","last_fullsized":"2024-03-26T21:22:09.840939+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:15:13.159606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840827+0000","last_change":"2024-03-26T21:21:25.188837+0000","last_active":"2024-03-26T21:22:09.840827+0000","last_peered":"2024-03-26T21:22:09.840827+0000","last_clean":"2024-03-26T21:22:09.840827+0000","last_became_active":"2024-03-26T21:21:25.188536+0000","last_became_peered":"2024-03-26T21:21:25.188536+0000","last_unstale":"2024-03-26T21:22:09.840827+0000","last_undegraded":"2024-03-26T21:22:09.840827+0000","last_fullsized":"2024-03-26T21:22:09.840827+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:30:45.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375647+0000","last_change":"2024-03-26T21:21:27.199658+0000","last_active":"2024-03-26T21:22:09.375647+0000","last_peered":"2024-03-26T21:22:09.375647+0000","last_clean":"2024-03-26T21:22:09.375647+0000","last_became_active":"2024-03-26T21:21:27.199336+0000","last_became_peered":"2024-03-26T21:21:27.199336+0000","last_unstale":"2024-03-26T21:22:09.375647+0000","last_undegraded":"2024-03-26T21:22:09.375647+0000","last_fullsized":"2024-03-26T21:22:09.375647+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:15:41.049691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248419+0000","last_change":"2024-03-26T21:21:29.209346+0000","last_active":"2024-03-26T21:22:40.248419+0000","last_peered":"2024-03-26T21:22:40.248419+0000","last_clean":"2024-03-26T21:22:40.248419+0000","last_became_active":"2024-03-26T21:21:29.209004+0000","last_became_peered":"2024-03-26T21:21:29.209004+0000","last_unstale":"2024-03-26T21:22:40.248419+0000","last_undegraded":"2024-03-26T21:22:40.248419+0000","last_fullsized":"2024-03-26T21:22:40.248419+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:22:33.834366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376493+0000","last_change":"2024-03-26T21:21:31.347339+0000","last_active":"2024-03-26T21:22:09.376493+0000","last_peered":"2024-03-26T21:22:09.376493+0000","last_clean":"2024-03-26T21:22:09.376493+0000","last_became_active":"2024-03-26T21:21:31.346912+0000","last_became_peered":"2024-03-26T21:21:31.346912+0000","last_unstale":"2024-03-26T21:22:09.376493+0000","last_undegraded":"2024-03-26T21:22:09.376493+0000","last_fullsized":"2024-03-26T21:22:09.376493+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:12:27.250572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840136+0000","last_change":"2024-03-26T21:21:29.198185+0000","last_active":"2024-03-26T21:22:09.840136+0000","last_peered":"2024-03-26T21:22:09.840136+0000","last_clean":"2024-03-26T21:22:09.840136+0000","last_became_active":"2024-03-26T21:21:29.197858+0000","last_became_peered":"2024-03-26T21:21:29.197858+0000","last_unstale":"2024-03-26T21:22:09.840136+0000","last_undegraded":"2024-03-26T21:22:09.840136+0000","last_fullsized":"2024-03-26T21:22:09.840136+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:10:29.651320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839375+0000","last_change":"2024-03-26T21:21:25.188797+0000","last_active":"2024-03-26T21:22:09.839375+0000","last_peered":"2024-03-26T21:22:09.839375+0000","last_clean":"2024-03-26T21:22:09.839375+0000","last_became_active":"2024-03-26T21:21:25.188492+0000","last_became_peered":"2024-03-26T21:21:25.188492+0000","last_unstale":"2024-03-26T21:22:09.839375+0000","last_undegraded":"2024-03-26T21:22:09.839375+0000","last_fullsized":"2024-03-26T21:22:09.839375+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:37:47.176328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376625+0000","last_change":"2024-03-26T21:21:27.206582+0000","last_active":"2024-03-26T21:22:09.376625+0000","last_peered":"2024-03-26T21:22:09.376625+0000","last_clean":"2024-03-26T21:22:09.376625+0000","last_became_active":"2024-03-26T21:21:27.206272+0000","last_became_peered":"2024-03-26T21:21:27.206272+0000","last_unstale":"2024-03-26T21:22:09.376625+0000","last_undegraded":"2024-03-26T21:22:09.376625+0000","last_fullsized":"2024-03-26T21:22:09.376625+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:19:03.157404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839290+0000","last_change":"2024-03-26T21:21:31.219142+0000","last_active":"2024-03-26T21:22:09.839290+0000","last_peered":"2024-03-26T21:22:09.839290+0000","last_clean":"2024-03-26T21:22:09.839290+0000","last_became_active":"2024-03-26T21:21:31.218915+0000","last_became_peered":"2024-03-26T21:21:31.218915+0000","last_unstale":"2024-03-26T21:22:09.839290+0000","last_undegraded":"2024-03-26T21:22:09.839290+0000","last_fullsized":"2024-03-26T21:22:09.839290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:12:30.083392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248823+0000","last_change":"2024-03-26T21:21:29.197672+0000","last_active":"2024-03-26T21:22:40.248823+0000","last_peered":"2024-03-26T21:22:40.248823+0000","last_clean":"2024-03-26T21:22:40.248823+0000","last_became_active":"2024-03-26T21:21:29.197295+0000","last_became_peered":"2024-03-26T21:21:29.197295+0000","last_unstale":"2024-03-26T21:22:40.248823+0000","last_undegraded":"2024-03-26T21:22:40.248823+0000","last_fullsized":"2024-03-26T21:22:40.248823+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:03:16.385162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376525+0000","last_change":"2024-03-26T21:21:25.193057+0000","last_active":"2024-03-26T21:22:09.376525+0000","last_peered":"2024-03-26T21:22:09.376525+0000","last_clean":"2024-03-26T21:22:09.376525+0000","last_became_active":"2024-03-26T21:21:25.192721+0000","last_became_peered":"2024-03-26T21:21:25.192721+0000","last_unstale":"2024-03-26T21:22:09.376525+0000","last_undegraded":"2024-03-26T21:22:09.376525+0000","last_fullsized":"2024-03-26T21:22:09.376525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:18:29.762531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.379273+0000","last_change":"2024-03-26T21:21:27.204279+0000","last_active":"2024-03-26T21:22:09.379273+0000","last_peered":"2024-03-26T21:22:09.379273+0000","last_clean":"2024-03-26T21:22:09.379273+0000","last_became_active":"2024-03-26T21:21:27.203993+0000","last_became_peered":"2024-03-26T21:21:27.203993+0000","last_unstale":"2024-03-26T21:22:09.379273+0000","last_undegraded":"2024-03-26T21:22:09.379273+0000","last_fullsized":"2024-03-26T21:22:09.379273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:11:44.363251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376343+0000","last_change":"2024-03-26T21:21:31.216339+0000","last_active":"2024-03-26T21:22:09.376343+0000","last_peered":"2024-03-26T21:22:09.376343+0000","last_clean":"2024-03-26T21:22:09.376343+0000","last_became_active":"2024-03-26T21:21:31.216056+0000","last_became_peered":"2024-03-26T21:21:31.216056+0000","last_unstale":"2024-03-26T21:22:09.376343+0000","last_undegraded":"2024-03-26T21:22:09.376343+0000","last_fullsized":"2024-03-26T21:22:09.376343+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:55:36.596444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248371+0000","last_change":"2024-03-26T21:21:29.202671+0000","last_active":"2024-03-26T21:22:40.248371+0000","last_peered":"2024-03-26T21:22:40.248371+0000","last_clean":"2024-03-26T21:22:40.248371+0000","last_became_active":"2024-03-26T21:21:29.202282+0000","last_became_peered":"2024-03-26T21:21:29.202282+0000","last_unstale":"2024-03-26T21:22:40.248371+0000","last_undegraded":"2024-03-26T21:22:40.248371+0000","last_fullsized":"2024-03-26T21:22:40.248371+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:37:25.062725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376920+0000","last_change":"2024-03-26T21:21:25.194419+0000","last_active":"2024-03-26T21:22:09.376920+0000","last_peered":"2024-03-26T21:22:09.376920+0000","last_clean":"2024-03-26T21:22:09.376920+0000","last_became_active":"2024-03-26T21:21:25.194067+0000","last_became_peered":"2024-03-26T21:21:25.194067+0000","last_unstale":"2024-03-26T21:22:09.376920+0000","last_undegraded":"2024-03-26T21:22:09.376920+0000","last_fullsized":"2024-03-26T21:22:09.376920+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:15:27.805137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375812+0000","last_change":"2024-03-26T21:21:27.207639+0000","last_active":"2024-03-26T21:22:09.375812+0000","last_peered":"2024-03-26T21:22:09.375812+0000","last_clean":"2024-03-26T21:22:09.375812+0000","last_became_active":"2024-03-26T21:21:27.207347+0000","last_became_peered":"2024-03-26T21:21:27.207347+0000","last_unstale":"2024-03-26T21:22:09.375812+0000","last_undegraded":"2024-03-26T21:22:09.375812+0000","last_fullsized":"2024-03-26T21:22:09.375812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:27:40.075866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838792+0000","last_change":"2024-03-26T21:21:31.217737+0000","last_active":"2024-03-26T21:22:09.838792+0000","last_peered":"2024-03-26T21:22:09.838792+0000","last_clean":"2024-03-26T21:22:09.838792+0000","last_became_active":"2024-03-26T21:21:31.217407+0000","last_became_peered":"2024-03-26T21:21:31.217407+0000","last_unstale":"2024-03-26T21:22:09.838792+0000","last_undegraded":"2024-03-26T21:22:09.838792+0000","last_fullsized":"2024-03-26T21:22:09.838792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:04:58.041327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377656+0000","last_change":"2024-03-26T21:21:29.206774+0000","last_active":"2024-03-26T21:22:09.377656+0000","last_peered":"2024-03-26T21:22:09.377656+0000","last_clean":"2024-03-26T21:22:09.377656+0000","last_became_active":"2024-03-26T21:21:29.206472+0000","last_became_peered":"2024-03-26T21:21:29.206472+0000","last_unstale":"2024-03-26T21:22:09.377656+0000","last_undegraded":"2024-03-26T21:22:09.377656+0000","last_fullsized":"2024-03-26T21:22:09.377656+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:52:43.335726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839303+0000","last_change":"2024-03-26T21:21:25.194241+0000","last_active":"2024-03-26T21:22:09.839303+0000","last_peered":"2024-03-26T21:22:09.839303+0000","last_clean":"2024-03-26T21:22:09.839303+0000","last_became_active":"2024-03-26T21:21:25.193964+0000","last_became_peered":"2024-03-26T21:21:25.193964+0000","last_unstale":"2024-03-26T21:22:09.839303+0000","last_undegraded":"2024-03-26T21:22:09.839303+0000","last_fullsized":"2024-03-26T21:22:09.839303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.520719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376132+0000","last_change":"2024-03-26T21:21:27.206502+0000","last_active":"2024-03-26T21:22:09.376132+0000","last_peered":"2024-03-26T21:22:09.376132+0000","last_clean":"2024-03-26T21:22:09.376132+0000","last_became_active":"2024-03-26T21:21:27.206173+0000","last_became_peered":"2024-03-26T21:21:27.206173+0000","last_unstale":"2024-03-26T21:22:09.376132+0000","last_undegraded":"2024-03-26T21:22:09.376132+0000","last_fullsized":"2024-03-26T21:22:09.376132+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:46:16.365138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839066+0000","last_change":"2024-03-26T21:21:31.209384+0000","last_active":"2024-03-26T21:22:09.839066+0000","last_peered":"2024-03-26T21:22:09.839066+0000","last_clean":"2024-03-26T21:22:09.839066+0000","last_became_active":"2024-03-26T21:21:31.209071+0000","last_became_peered":"2024-03-26T21:21:31.209071+0000","last_unstale":"2024-03-26T21:22:09.839066+0000","last_undegraded":"2024-03-26T21:22:09.839066+0000","last_fullsized":"2024-03-26T21:22:09.839066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:56:14.900945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838498+0000","last_change":"2024-03-26T21:21:29.200667+0000","last_active":"2024-03-26T21:22:09.838498+0000","last_peered":"2024-03-26T21:22:09.838498+0000","last_clean":"2024-03-26T21:22:09.838498+0000","last_became_active":"2024-03-26T21:21:29.200345+0000","last_became_peered":"2024-03-26T21:21:29.200345+0000","last_unstale":"2024-03-26T21:22:09.838498+0000","last_undegraded":"2024-03-26T21:22:09.838498+0000","last_fullsized":"2024-03-26T21:22:09.838498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:20:43.952201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838575+0000","last_change":"2024-03-26T21:21:25.188968+0000","last_active":"2024-03-26T21:22:09.838575+0000","last_peered":"2024-03-26T21:22:09.838575+0000","last_clean":"2024-03-26T21:22:09.838575+0000","last_became_active":"2024-03-26T21:21:25.188613+0000","last_became_peered":"2024-03-26T21:21:25.188613+0000","last_unstale":"2024-03-26T21:22:09.838575+0000","last_undegraded":"2024-03-26T21:22:09.838575+0000","last_fullsized":"2024-03-26T21:22:09.838575+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:01:31.748061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839029+0000","last_change":"2024-03-26T21:21:27.196031+0000","last_active":"2024-03-26T21:22:09.839029+0000","last_peered":"2024-03-26T21:22:09.839029+0000","last_clean":"2024-03-26T21:22:09.839029+0000","last_became_active":"2024-03-26T21:21:27.195718+0000","last_became_peered":"2024-03-26T21:21:27.195718+0000","last_unstale":"2024-03-26T21:22:09.839029+0000","last_undegraded":"2024-03-26T21:22:09.839029+0000","last_fullsized":"2024-03-26T21:22:09.839029+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:30:49.939984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494091+0000","last_change":"2024-03-26T21:21:31.210232+0000","last_active":"2024-03-26T21:22:09.494091+0000","last_peered":"2024-03-26T21:22:09.494091+0000","last_clean":"2024-03-26T21:22:09.494091+0000","last_became_active":"2024-03-26T21:21:31.209869+0000","last_became_peered":"2024-03-26T21:21:31.209869+0000","last_unstale":"2024-03-26T21:22:09.494091+0000","last_undegraded":"2024-03-26T21:22:09.494091+0000","last_fullsized":"2024-03-26T21:22:09.494091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:35:50.979234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838394+0000","last_change":"2024-03-26T21:21:29.199110+0000","last_active":"2024-03-26T21:22:09.838394+0000","last_peered":"2024-03-26T21:22:09.838394+0000","last_clean":"2024-03-26T21:22:09.838394+0000","last_became_active":"2024-03-26T21:21:29.198790+0000","last_became_peered":"2024-03-26T21:21:29.198790+0000","last_unstale":"2024-03-26T21:22:09.838394+0000","last_undegraded":"2024-03-26T21:22:09.838394+0000","last_fullsized":"2024-03-26T21:22:09.838394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:01:02.970776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378425+0000","last_change":"2024-03-26T21:21:25.193996+0000","last_active":"2024-03-26T21:22:09.378425+0000","last_peered":"2024-03-26T21:22:09.378425+0000","last_clean":"2024-03-26T21:22:09.378425+0000","last_became_active":"2024-03-26T21:21:25.192736+0000","last_became_peered":"2024-03-26T21:21:25.192736+0000","last_unstale":"2024-03-26T21:22:09.378425+0000","last_undegraded":"2024-03-26T21:22:09.378425+0000","last_fullsized":"2024-03-26T21:22:09.378425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:06:45.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494955+0000","last_change":"2024-03-26T21:21:27.195683+0000","last_active":"2024-03-26T21:22:09.494955+0000","last_peered":"2024-03-26T21:22:09.494955+0000","last_clean":"2024-03-26T21:22:09.494955+0000","last_became_active":"2024-03-26T21:21:27.195230+0000","last_became_peered":"2024-03-26T21:21:27.195230+0000","last_unstale":"2024-03-26T21:22:09.494955+0000","last_undegraded":"2024-03-26T21:22:09.494955+0000","last_fullsized":"2024-03-26T21:22:09.494955+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:35:37.322721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838858+0000","last_change":"2024-03-26T21:21:31.347442+0000","last_active":"2024-03-26T21:22:09.838858+0000","last_peered":"2024-03-26T21:22:09.838858+0000","last_clean":"2024-03-26T21:22:09.838858+0000","last_became_active":"2024-03-26T21:21:31.347054+0000","last_became_peered":"2024-03-26T21:21:31.347054+0000","last_unstale":"2024-03-26T21:22:09.838858+0000","last_undegraded":"2024-03-26T21:22:09.838858+0000","last_fullsized":"2024-03-26T21:22:09.838858+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:02:10.676163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838823+0000","last_change":"2024-03-26T21:21:25.188916+0000","last_active":"2024-03-26T21:22:09.838823+0000","last_peered":"2024-03-26T21:22:09.838823+0000","last_clean":"2024-03-26T21:22:09.838823+0000","last_became_active":"2024-03-26T21:21:25.188139+0000","last_became_peered":"2024-03-26T21:21:25.188139+0000","last_unstale":"2024-03-26T21:22:09.838823+0000","last_undegraded":"2024-03-26T21:22:09.838823+0000","last_fullsized":"2024-03-26T21:22:09.838823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:49:38.834721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838510+0000","last_change":"2024-03-26T21:21:27.194089+0000","last_active":"2024-03-26T21:22:09.838510+0000","last_peered":"2024-03-26T21:22:09.838510+0000","last_clean":"2024-03-26T21:22:09.838510+0000","last_became_active":"2024-03-26T21:21:27.193627+0000","last_became_peered":"2024-03-26T21:21:27.193627+0000","last_unstale":"2024-03-26T21:22:09.838510+0000","last_undegraded":"2024-03-26T21:22:09.838510+0000","last_fullsized":"2024-03-26T21:22:09.838510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:40:04.185496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:40.248827+0000","last_change":"2024-03-26T21:21:29.201941+0000","last_active":"2024-03-26T21:22:40.248827+0000","last_peered":"2024-03-26T21:22:40.248827+0000","last_clean":"2024-03-26T21:22:40.248827+0000","last_became_active":"2024-03-26T21:21:29.201651+0000","last_became_peered":"2024-03-26T21:21:29.201651+0000","last_unstale":"2024-03-26T21:22:40.248827+0000","last_undegraded":"2024-03-26T21:22:40.248827+0000","last_fullsized":"2024-03-26T21:22:40.248827+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:09:54.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1298432,"data_stored":867823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1294336,"data_stored":868116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":741376,"data_stored":278780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1298432,"data_stored":867129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":692224,"data_stored":276983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":277702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":276928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31864,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713288,"statfs":{"total":95995035648,"available":95962406912,"internally_reserved":0,"allocated":700416,"data_stored":278139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:22:47.010 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:22:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:47 smithi088 ceph-mon[39202]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:47.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:47 smithi088 ceph-mon[30075]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:47.258 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:47 smithi138 ceph-mon[33706]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:48 smithi138 ceph-mon[33706]: from='client.34634 v1:172.21.15.88:0/2153052322' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:22:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:48 smithi088 ceph-mon[39202]: from='client.34634 v1:172.21.15.88:0/2153052322' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:22:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:48 smithi088 ceph-mon[30075]: from='client.34634 v1:172.21.15.88:0/2153052322' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:22:49.249 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:49 smithi088 ceph-mon[30075]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:49.249 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:49 smithi088 ceph-mon[39202]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:49 smithi138 ceph-mon[33706]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:50.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:22:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:22:51.336 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:51 smithi088 ceph-mon[39202]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:51.336 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:51 smithi088 ceph-mon[30075]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:51 smithi138 ceph-mon[33706]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:51.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.833+0000 7f7ebdef8700 1 -- 172.21.15.88:0/2461499524 >> v1:172.21.15.88:6790/0 conn(0x7f7eb810d550 legacy=0x7f7eb810fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:51.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.833+0000 7f7ebdef8700 1 -- 172.21.15.88:0/2461499524 shutdown_connections 2024-03-26T21:22:51.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.833+0000 7f7ebdef8700 1 -- 172.21.15.88:0/2461499524 >> 172.21.15.88:0/2461499524 conn(0x7f7eb80765c0 msgr2=0x7f7eb80769d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.833+0000 7f7ebdef8700 1 -- 172.21.15.88:0/2461499524 shutdown_connections 2024-03-26T21:22:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.833+0000 7f7ebdef8700 1 -- 172.21.15.88:0/2461499524 wait complete. 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.833+0000 7f7ebdef8700 1 Processor -- start 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7ebdef8700 1 -- start start 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7ebdef8700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7eb81a9780 con 0x7f7eb81098f0 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7ebdef8700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7eb81aa9a0 con 0x7f7eb810d550 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7ebdef8700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7eb81abbe0 con 0x7f7eb8106ab0 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb77fe700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f7eb8106ab0 0x7f7eb8104aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:57486/0 (socket says 172.21.15.88:57486) 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb77fe700 1 -- 172.21.15.88:0/3394873353 learned_addr learned my addr 172.21.15.88:0/3394873353 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717392757 0 0) 0x7f7eb81abbe0 con 0x7f7eb8106ab0 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ea00046c0 con 0x7f7eb8106ab0 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870202332 0 0) 0x7f7eb81a9780 con 0x7f7eb81098f0 2024-03-26T21:22:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb81abbe0 con 0x7f7eb81098f0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 137638315 0 0) 0x7f7eb81aa9a0 con 0x7f7eb810d550 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb81a9780 con 0x7f7eb810d550 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2867368296 0 0) 0x7f7ea00046c0 con 0x7f7eb8106ab0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eb81aa9a0 con 0x7f7eb8106ab0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3260274066 0 0) 0x7f7eb81abbe0 con 0x7f7eb81098f0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ea00046c0 con 0x7f7eb81098f0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7eac0010f0 con 0x7f7eb81098f0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7ea8001930 con 0x7f7eb8106ab0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1385062132 0 0) 0x7f7eb81a9780 con 0x7f7eb810d550 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eb81abbe0 con 0x7f7eb810d550 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.834+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7eb00046f0 con 0x7f7eb810d550 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4054781168 0 0) 0x7f7ea00046c0 con 0x7f7eb81098f0 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 >> v1:172.21.15.88:6790/0 conn(0x7f7eb8106ab0 legacy=0x7f7eb8104aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 >> v1:172.21.15.138:6789/0 conn(0x7f7eb810d550 legacy=0x7f7eb81a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb81ace20 con 0x7f7eb81098f0 2024-03-26T21:22:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb81a99b0 con 0x7f7eb81098f0 2024-03-26T21:22:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7eb81a9e70 con 0x7f7eb81098f0 2024-03-26T21:22:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f7eac003b10 con 0x7f7eb81098f0 2024-03-26T21:22:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.835+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7eac004a30 con 0x7f7eb81098f0 2024-03-26T21:22:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.836+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea4005120 con 0x7f7eb81098f0 2024-03-26T21:22:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.836+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f7eac003680 con 0x7f7eb81098f0 2024-03-26T21:22:51.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.837+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f7eac088a40 con 0x7f7eb81098f0 2024-03-26T21:22:51.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:51.841+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f7eac055ee0 con 0x7f7eb81098f0 2024-03-26T21:22:52.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.042+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 --> v1:172.21.15.88:6800/3939589528 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7ea4000b90 con 0x7f7ea006f630 2024-03-26T21:22:52.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.052+0000 7f7eb4ff9700 1 -- 172.21.15.88:0/3394873353 <== mgr.14622 v1:172.21.15.88:6800/3939589528 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346463 (unknown 2965378022 0 420591138) 0x7f7ea4000b90 con 0x7f7ea006f630 2024-03-26T21:22:52.049 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:22:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.055+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 >> v1:172.21.15.88:6800/3939589528 conn(0x7f7ea006f630 legacy=0x7f7ea0071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.055+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 >> v1:172.21.15.88:6789/0 conn(0x7f7eb81098f0 legacy=0x7f7eb81051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.055+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 shutdown_connections 2024-03-26T21:22:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.055+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 >> 172.21.15.88:0/3394873353 conn(0x7f7eb80765c0 msgr2=0x7f7eb81080c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.055+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 shutdown_connections 2024-03-26T21:22:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:52.055+0000 7f7ebdef8700 1 -- 172.21.15.88:0/3394873353 wait complete. 2024-03-26T21:22:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-03-26T21:22:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:22:52.568 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:22:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:22:53.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:53 smithi088 ceph-mon[30075]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:53.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:53 smithi088 ceph-mon[30075]: from='client.14760 v1:172.21.15.88:0/3394873353' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:22:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:53 smithi088 ceph-mon[39202]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:53 smithi088 ceph-mon[39202]: from='client.14760 v1:172.21.15.88:0/3394873353' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:22:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:53 smithi138 ceph-mon[33706]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:53 smithi138 ceph-mon[33706]: from='client.14760 v1:172.21.15.88:0/3394873353' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:22:53.734 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:53 smithi088 systemd[1]: Starting Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:22:53.785 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-26T21:22:51.393265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1013,"num_read_kb":859,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235932,"kb_used_data":7292,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725284,"statfs":{"total":767960285184,"available":767718690816,"internally_reserved":0,"allocated":7467008,"data_stored":3991600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002618"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374766+0000","last_change":"2024-03-26T21:21:31.349104+0000","last_active":"2024-03-26T21:22:09.374766+0000","last_peered":"2024-03-26T21:22:09.374766+0000","last_clean":"2024-03-26T21:22:09.374766+0000","last_became_active":"2024-03-26T21:21:31.348758+0000","last_became_peered":"2024-03-26T21:21:31.348758+0000","last_unstale":"2024-03-26T21:22:09.374766+0000","last_undegraded":"2024-03-26T21:22:09.374766+0000","last_fullsized":"2024-03-26T21:22:09.374766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:32:54.707488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375349+0000","last_change":"2024-03-26T21:21:29.206796+0000","last_active":"2024-03-26T21:22:09.375349+0000","last_peered":"2024-03-26T21:22:09.375349+0000","last_clean":"2024-03-26T21:22:09.375349+0000","last_became_active":"2024-03-26T21:21:29.206440+0000","last_became_peered":"2024-03-26T21:21:29.206440+0000","last_unstale":"2024-03-26T21:22:09.375349+0000","last_undegraded":"2024-03-26T21:22:09.375349+0000","last_fullsized":"2024-03-26T21:22:09.375349+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:03:26.449042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494434+0000","last_change":"2024-03-26T21:21:25.189507+0000","last_active":"2024-03-26T21:22:09.494434+0000","last_peered":"2024-03-26T21:22:09.494434+0000","last_clean":"2024-03-26T21:22:09.494434+0000","last_became_active":"2024-03-26T21:21:25.189192+0000","last_became_peered":"2024-03-26T21:21:25.189192+0000","last_unstale":"2024-03-26T21:22:09.494434+0000","last_undegraded":"2024-03-26T21:22:09.494434+0000","last_fullsized":"2024-03-26T21:22:09.494434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:24:36.169921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374670+0000","last_change":"2024-03-26T21:21:27.201221+0000","last_active":"2024-03-26T21:22:09.374670+0000","last_peered":"2024-03-26T21:22:09.374670+0000","last_clean":"2024-03-26T21:22:09.374670+0000","last_became_active":"2024-03-26T21:21:27.200913+0000","last_became_peered":"2024-03-26T21:21:27.200913+0000","last_unstale":"2024-03-26T21:22:09.374670+0000","last_undegraded":"2024-03-26T21:22:09.374670+0000","last_fullsized":"2024-03-26T21:22:09.374670+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:39:17.901410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375532+0000","last_change":"2024-03-26T21:21:31.216461+0000","last_active":"2024-03-26T21:22:09.375532+0000","last_peered":"2024-03-26T21:22:09.375532+0000","last_clean":"2024-03-26T21:22:09.375532+0000","last_became_active":"2024-03-26T21:21:31.216128+0000","last_became_peered":"2024-03-26T21:21:31.216128+0000","last_unstale":"2024-03-26T21:22:09.375532+0000","last_undegraded":"2024-03-26T21:22:09.375532+0000","last_fullsized":"2024-03-26T21:22:09.375532+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:47:19.990154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376376+0000","last_change":"2024-03-26T21:21:29.206305+0000","last_active":"2024-03-26T21:22:09.376376+0000","last_peered":"2024-03-26T21:22:09.376376+0000","last_clean":"2024-03-26T21:22:09.376376+0000","last_became_active":"2024-03-26T21:21:29.205975+0000","last_became_peered":"2024-03-26T21:21:29.205975+0000","last_unstale":"2024-03-26T21:22:09.376376+0000","last_undegraded":"2024-03-26T21:22:09.376376+0000","last_fullsized":"2024-03-26T21:22:09.376376+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:26:09.381171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375686+0000","last_change":"2024-03-26T21:21:25.189428+0000","last_active":"2024-03-26T21:22:09.375686+0000","last_peered":"2024-03-26T21:22:09.375686+0000","last_clean":"2024-03-26T21:22:09.375686+0000","last_became_active":"2024-03-26T21:21:25.189152+0000","last_became_peered":"2024-03-26T21:21:25.189152+0000","last_unstale":"2024-03-26T21:22:09.375686+0000","last_undegraded":"2024-03-26T21:22:09.375686+0000","last_fullsized":"2024-03-26T21:22:09.375686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:57:13.834170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375492+0000","last_change":"2024-03-26T21:21:27.206416+0000","last_active":"2024-03-26T21:22:09.375492+0000","last_peered":"2024-03-26T21:22:09.375492+0000","last_clean":"2024-03-26T21:22:09.375492+0000","last_became_active":"2024-03-26T21:21:27.206108+0000","last_became_peered":"2024-03-26T21:21:27.206108+0000","last_unstale":"2024-03-26T21:22:09.375492+0000","last_undegraded":"2024-03-26T21:22:09.375492+0000","last_fullsized":"2024-03-26T21:22:09.375492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:16:20.138623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838517+0000","last_change":"2024-03-26T21:21:31.215674+0000","last_active":"2024-03-26T21:22:09.838517+0000","last_peered":"2024-03-26T21:22:09.838517+0000","last_clean":"2024-03-26T21:22:09.838517+0000","last_became_active":"2024-03-26T21:21:31.215409+0000","last_became_peered":"2024-03-26T21:21:31.215409+0000","last_unstale":"2024-03-26T21:22:09.838517+0000","last_undegraded":"2024-03-26T21:22:09.838517+0000","last_fullsized":"2024-03-26T21:22:09.838517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:36:38.087934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840290+0000","last_change":"2024-03-26T21:21:29.204643+0000","last_active":"2024-03-26T21:22:09.840290+0000","last_peered":"2024-03-26T21:22:09.840290+0000","last_clean":"2024-03-26T21:22:09.840290+0000","last_became_active":"2024-03-26T21:21:29.204343+0000","last_became_peered":"2024-03-26T21:21:29.204343+0000","last_unstale":"2024-03-26T21:22:09.840290+0000","last_undegraded":"2024-03-26T21:22:09.840290+0000","last_fullsized":"2024-03-26T21:22:09.840290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:39:36.717265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840336+0000","last_change":"2024-03-26T21:21:25.190623+0000","last_active":"2024-03-26T21:22:09.840336+0000","last_peered":"2024-03-26T21:22:09.840336+0000","last_clean":"2024-03-26T21:22:09.840336+0000","last_became_active":"2024-03-26T21:21:25.190084+0000","last_became_peered":"2024-03-26T21:21:25.190084+0000","last_unstale":"2024-03-26T21:22:09.840336+0000","last_undegraded":"2024-03-26T21:22:09.840336+0000","last_fullsized":"2024-03-26T21:22:09.840336+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:24:42.314966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376172+0000","last_change":"2024-03-26T21:21:48.190967+0000","last_active":"2024-03-26T21:22:09.376172+0000","last_peered":"2024-03-26T21:22:09.376172+0000","last_clean":"2024-03-26T21:22:09.376172+0000","last_became_active":"2024-03-26T21:21:48.177524+0000","last_became_peered":"2024-03-26T21:21:48.177524+0000","last_unstale":"2024-03-26T21:22:09.376172+0000","last_undegraded":"2024-03-26T21:22:09.376172+0000","last_fullsized":"2024-03-26T21:22:09.376172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:33:07.643016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494368+0000","last_change":"2024-03-26T21:21:31.219439+0000","last_active":"2024-03-26T21:22:09.494368+0000","last_peered":"2024-03-26T21:22:09.494368+0000","last_clean":"2024-03-26T21:22:09.494368+0000","last_became_active":"2024-03-26T21:21:31.219016+0000","last_became_peered":"2024-03-26T21:21:31.219016+0000","last_unstale":"2024-03-26T21:22:09.494368+0000","last_undegraded":"2024-03-26T21:22:09.494368+0000","last_fullsized":"2024-03-26T21:22:09.494368+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:04:29.242885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840018+0000","last_change":"2024-03-26T21:21:29.197106+0000","last_active":"2024-03-26T21:22:09.840018+0000","last_peered":"2024-03-26T21:22:09.840018+0000","last_clean":"2024-03-26T21:22:09.840018+0000","last_became_active":"2024-03-26T21:21:29.196780+0000","last_became_peered":"2024-03-26T21:21:29.196780+0000","last_unstale":"2024-03-26T21:22:09.840018+0000","last_undegraded":"2024-03-26T21:22:09.840018+0000","last_fullsized":"2024-03-26T21:22:09.840018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:34:33.394076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839675+0000","last_change":"2024-03-26T21:21:25.193368+0000","last_active":"2024-03-26T21:22:09.839675+0000","last_peered":"2024-03-26T21:22:09.839675+0000","last_clean":"2024-03-26T21:22:09.839675+0000","last_became_active":"2024-03-26T21:21:25.193073+0000","last_became_peered":"2024-03-26T21:21:25.193073+0000","last_unstale":"2024-03-26T21:22:09.839675+0000","last_undegraded":"2024-03-26T21:22:09.839675+0000","last_fullsized":"2024-03-26T21:22:09.839675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:53:30.706559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375554+0000","last_change":"2024-03-26T21:21:27.208458+0000","last_active":"2024-03-26T21:22:09.375554+0000","last_peered":"2024-03-26T21:22:09.375554+0000","last_clean":"2024-03-26T21:22:09.375554+0000","last_became_active":"2024-03-26T21:21:27.208126+0000","last_became_peered":"2024-03-26T21:21:27.208126+0000","last_unstale":"2024-03-26T21:22:09.375554+0000","last_undegraded":"2024-03-26T21:22:09.375554+0000","last_fullsized":"2024-03-26T21:22:09.375554+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:47:19.052796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376161+0000","last_change":"2024-03-26T21:21:31.349018+0000","last_active":"2024-03-26T21:22:09.376161+0000","last_peered":"2024-03-26T21:22:09.376161+0000","last_clean":"2024-03-26T21:22:09.376161+0000","last_became_active":"2024-03-26T21:21:31.348709+0000","last_became_peered":"2024-03-26T21:21:31.348709+0000","last_unstale":"2024-03-26T21:22:09.376161+0000","last_undegraded":"2024-03-26T21:22:09.376161+0000","last_fullsized":"2024-03-26T21:22:09.376161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:04:23.573573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375942+0000","last_change":"2024-03-26T21:21:29.203037+0000","last_active":"2024-03-26T21:22:09.375942+0000","last_peered":"2024-03-26T21:22:09.375942+0000","last_clean":"2024-03-26T21:22:09.375942+0000","last_became_active":"2024-03-26T21:21:29.202728+0000","last_became_peered":"2024-03-26T21:21:29.202728+0000","last_unstale":"2024-03-26T21:22:09.375942+0000","last_undegraded":"2024-03-26T21:22:09.375942+0000","last_fullsized":"2024-03-26T21:22:09.375942+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:49:46.677343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375235+0000","last_change":"2024-03-26T21:21:25.188941+0000","last_active":"2024-03-26T21:22:09.375235+0000","last_peered":"2024-03-26T21:22:09.375235+0000","last_clean":"2024-03-26T21:22:09.375235+0000","last_became_active":"2024-03-26T21:21:25.188559+0000","last_became_peered":"2024-03-26T21:21:25.188559+0000","last_unstale":"2024-03-26T21:22:09.375235+0000","last_undegraded":"2024-03-26T21:22:09.375235+0000","last_fullsized":"2024-03-26T21:22:09.375235+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:46:08.987380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375163+0000","last_change":"2024-03-26T21:21:27.202424+0000","last_active":"2024-03-26T21:22:09.375163+0000","last_peered":"2024-03-26T21:22:09.375163+0000","last_clean":"2024-03-26T21:22:09.375163+0000","last_became_active":"2024-03-26T21:21:27.202113+0000","last_became_peered":"2024-03-26T21:21:27.202113+0000","last_unstale":"2024-03-26T21:22:09.375163+0000","last_undegraded":"2024-03-26T21:22:09.375163+0000","last_fullsized":"2024-03-26T21:22:09.375163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:25:50.165698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376528+0000","last_change":"2024-03-26T21:21:31.348441+0000","last_active":"2024-03-26T21:22:09.376528+0000","last_peered":"2024-03-26T21:22:09.376528+0000","last_clean":"2024-03-26T21:22:09.376528+0000","last_became_active":"2024-03-26T21:21:31.348124+0000","last_became_peered":"2024-03-26T21:21:31.348124+0000","last_unstale":"2024-03-26T21:22:09.376528+0000","last_undegraded":"2024-03-26T21:22:09.376528+0000","last_fullsized":"2024-03-26T21:22:09.376528+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:48:58.772269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378559+0000","last_change":"2024-03-26T21:21:29.207591+0000","last_active":"2024-03-26T21:22:09.378559+0000","last_peered":"2024-03-26T21:22:09.378559+0000","last_clean":"2024-03-26T21:22:09.378559+0000","last_became_active":"2024-03-26T21:21:29.207132+0000","last_became_peered":"2024-03-26T21:21:29.207132+0000","last_unstale":"2024-03-26T21:22:09.378559+0000","last_undegraded":"2024-03-26T21:22:09.378559+0000","last_fullsized":"2024-03-26T21:22:09.378559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:41:23.529202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839178+0000","last_change":"2024-03-26T21:21:25.190870+0000","last_active":"2024-03-26T21:22:09.839178+0000","last_peered":"2024-03-26T21:22:09.839178+0000","last_clean":"2024-03-26T21:22:09.839178+0000","last_became_active":"2024-03-26T21:21:25.190522+0000","last_became_peered":"2024-03-26T21:21:25.190522+0000","last_unstale":"2024-03-26T21:22:09.839178+0000","last_undegraded":"2024-03-26T21:22:09.839178+0000","last_fullsized":"2024-03-26T21:22:09.839178+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:10:37.983490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838892+0000","last_change":"2024-03-26T21:21:27.203220+0000","last_active":"2024-03-26T21:22:09.838892+0000","last_peered":"2024-03-26T21:22:09.838892+0000","last_clean":"2024-03-26T21:22:09.838892+0000","last_became_active":"2024-03-26T21:21:27.202842+0000","last_became_peered":"2024-03-26T21:21:27.202842+0000","last_unstale":"2024-03-26T21:22:09.838892+0000","last_undegraded":"2024-03-26T21:22:09.838892+0000","last_fullsized":"2024-03-26T21:22:09.838892+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:44:39.557554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378097+0000","last_change":"2024-03-26T21:21:31.218162+0000","last_active":"2024-03-26T21:22:09.378097+0000","last_peered":"2024-03-26T21:22:09.378097+0000","last_clean":"2024-03-26T21:22:09.378097+0000","last_became_active":"2024-03-26T21:21:31.217319+0000","last_became_peered":"2024-03-26T21:21:31.217319+0000","last_unstale":"2024-03-26T21:22:09.378097+0000","last_undegraded":"2024-03-26T21:22:09.378097+0000","last_fullsized":"2024-03-26T21:22:09.378097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:03:05.033927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.495053+0000","last_change":"2024-03-26T21:21:29.197343+0000","last_active":"2024-03-26T21:22:09.495053+0000","last_peered":"2024-03-26T21:22:09.495053+0000","last_clean":"2024-03-26T21:22:09.495053+0000","last_became_active":"2024-03-26T21:21:29.197027+0000","last_became_peered":"2024-03-26T21:21:29.197027+0000","last_unstale":"2024-03-26T21:22:09.495053+0000","last_undegraded":"2024-03-26T21:22:09.495053+0000","last_fullsized":"2024-03-26T21:22:09.495053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:53:35.883369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374677+0000","last_change":"2024-03-26T21:21:25.187709+0000","last_active":"2024-03-26T21:22:09.374677+0000","last_peered":"2024-03-26T21:22:09.374677+0000","last_clean":"2024-03-26T21:22:09.374677+0000","last_became_active":"2024-03-26T21:21:25.187425+0000","last_became_peered":"2024-03-26T21:21:25.187425+0000","last_unstale":"2024-03-26T21:22:09.374677+0000","last_undegraded":"2024-03-26T21:22:09.374677+0000","last_fullsized":"2024-03-26T21:22:09.374677+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:50:23.189954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840644+0000","last_change":"2024-03-26T21:21:27.198028+0000","last_active":"2024-03-26T21:22:09.840644+0000","last_peered":"2024-03-26T21:22:09.840644+0000","last_clean":"2024-03-26T21:22:09.840644+0000","last_became_active":"2024-03-26T21:21:27.197726+0000","last_became_peered":"2024-03-26T21:21:27.197726+0000","last_unstale":"2024-03-26T21:22:09.840644+0000","last_undegraded":"2024-03-26T21:22:09.840644+0000","last_fullsized":"2024-03-26T21:22:09.840644+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:07:21.862157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840163+0000","last_change":"2024-03-26T21:21:31.213514+0000","last_active":"2024-03-26T21:22:09.840163+0000","last_peered":"2024-03-26T21:22:09.840163+0000","last_clean":"2024-03-26T21:22:09.840163+0000","last_became_active":"2024-03-26T21:21:31.212790+0000","last_became_peered":"2024-03-26T21:21:31.212790+0000","last_unstale":"2024-03-26T21:22:09.840163+0000","last_undegraded":"2024-03-26T21:22:09.840163+0000","last_fullsized":"2024-03-26T21:22:09.840163+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:59:43.621549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249886+0000","last_change":"2024-03-26T21:21:29.198690+0000","last_active":"2024-03-26T21:22:45.249886+0000","last_peered":"2024-03-26T21:22:45.249886+0000","last_clean":"2024-03-26T21:22:45.249886+0000","last_became_active":"2024-03-26T21:21:29.198351+0000","last_became_peered":"2024-03-26T21:21:29.198351+0000","last_unstale":"2024-03-26T21:22:45.249886+0000","last_undegraded":"2024-03-26T21:22:45.249886+0000","last_fullsized":"2024-03-26T21:22:45.249886+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:33:43.550517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838403+0000","last_change":"2024-03-26T21:21:25.196429+0000","last_active":"2024-03-26T21:22:09.838403+0000","last_peered":"2024-03-26T21:22:09.838403+0000","last_clean":"2024-03-26T21:22:09.838403+0000","last_became_active":"2024-03-26T21:21:25.195391+0000","last_became_peered":"2024-03-26T21:21:25.195391+0000","last_unstale":"2024-03-26T21:22:09.838403+0000","last_undegraded":"2024-03-26T21:22:09.838403+0000","last_fullsized":"2024-03-26T21:22:09.838403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:36:31.294183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494920+0000","last_change":"2024-03-26T21:21:27.200636+0000","last_active":"2024-03-26T21:22:09.494920+0000","last_peered":"2024-03-26T21:22:09.494920+0000","last_clean":"2024-03-26T21:22:09.494920+0000","last_became_active":"2024-03-26T21:21:27.200174+0000","last_became_peered":"2024-03-26T21:21:27.200174+0000","last_unstale":"2024-03-26T21:22:09.494920+0000","last_undegraded":"2024-03-26T21:22:09.494920+0000","last_fullsized":"2024-03-26T21:22:09.494920+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:18:28.643508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375483+0000","last_change":"2024-03-26T21:21:31.347515+0000","last_active":"2024-03-26T21:22:09.375483+0000","last_peered":"2024-03-26T21:22:09.375483+0000","last_clean":"2024-03-26T21:22:09.375483+0000","last_became_active":"2024-03-26T21:21:31.347207+0000","last_became_peered":"2024-03-26T21:21:31.347207+0000","last_unstale":"2024-03-26T21:22:09.375483+0000","last_undegraded":"2024-03-26T21:22:09.375483+0000","last_fullsized":"2024-03-26T21:22:09.375483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:17:27.787975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839583+0000","last_change":"2024-03-26T21:21:29.202118+0000","last_active":"2024-03-26T21:22:09.839583+0000","last_peered":"2024-03-26T21:22:09.839583+0000","last_clean":"2024-03-26T21:22:09.839583+0000","last_became_active":"2024-03-26T21:21:29.201813+0000","last_became_peered":"2024-03-26T21:21:29.201813+0000","last_unstale":"2024-03-26T21:22:09.839583+0000","last_undegraded":"2024-03-26T21:22:09.839583+0000","last_fullsized":"2024-03-26T21:22:09.839583+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:24:21.520611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840885+0000","last_change":"2024-03-26T21:21:25.194344+0000","last_active":"2024-03-26T21:22:09.840885+0000","last_peered":"2024-03-26T21:22:09.840885+0000","last_clean":"2024-03-26T21:22:09.840885+0000","last_became_active":"2024-03-26T21:21:25.194061+0000","last_became_peered":"2024-03-26T21:21:25.194061+0000","last_unstale":"2024-03-26T21:22:09.840885+0000","last_undegraded":"2024-03-26T21:22:09.840885+0000","last_fullsized":"2024-03-26T21:22:09.840885+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:34:47.923066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374962+0000","last_change":"2024-03-26T21:21:27.205170+0000","last_active":"2024-03-26T21:22:09.374962+0000","last_peered":"2024-03-26T21:22:09.374962+0000","last_clean":"2024-03-26T21:22:09.374962+0000","last_became_active":"2024-03-26T21:21:27.203150+0000","last_became_peered":"2024-03-26T21:21:27.203150+0000","last_unstale":"2024-03-26T21:22:09.374962+0000","last_undegraded":"2024-03-26T21:22:09.374962+0000","last_fullsized":"2024-03-26T21:22:09.374962+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:49:06.759997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839396+0000","last_change":"2024-03-26T21:21:31.212695+0000","last_active":"2024-03-26T21:22:09.839396+0000","last_peered":"2024-03-26T21:22:09.839396+0000","last_clean":"2024-03-26T21:22:09.839396+0000","last_became_active":"2024-03-26T21:21:31.212361+0000","last_became_peered":"2024-03-26T21:21:31.212361+0000","last_unstale":"2024-03-26T21:22:09.839396+0000","last_undegraded":"2024-03-26T21:22:09.839396+0000","last_fullsized":"2024-03-26T21:22:09.839396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:43:43.165565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840565+0000","last_change":"2024-03-26T21:21:29.200516+0000","last_active":"2024-03-26T21:22:09.840565+0000","last_peered":"2024-03-26T21:22:09.840565+0000","last_clean":"2024-03-26T21:22:09.840565+0000","last_became_active":"2024-03-26T21:21:29.200203+0000","last_became_peered":"2024-03-26T21:21:29.200203+0000","last_unstale":"2024-03-26T21:22:09.840565+0000","last_undegraded":"2024-03-26T21:22:09.840565+0000","last_fullsized":"2024-03-26T21:22:09.840565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:57:23.985115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839817+0000","last_change":"2024-03-26T21:21:25.194146+0000","last_active":"2024-03-26T21:22:09.839817+0000","last_peered":"2024-03-26T21:22:09.839817+0000","last_clean":"2024-03-26T21:22:09.839817+0000","last_became_active":"2024-03-26T21:21:25.193199+0000","last_became_peered":"2024-03-26T21:21:25.193199+0000","last_unstale":"2024-03-26T21:22:09.839817+0000","last_undegraded":"2024-03-26T21:22:09.839817+0000","last_fullsized":"2024-03-26T21:22:09.839817+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:50:20.411801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374611+0000","last_change":"2024-03-26T21:21:27.203002+0000","last_active":"2024-03-26T21:22:09.374611+0000","last_peered":"2024-03-26T21:22:09.374611+0000","last_clean":"2024-03-26T21:22:09.374611+0000","last_became_active":"2024-03-26T21:21:27.202176+0000","last_became_peered":"2024-03-26T21:21:27.202176+0000","last_unstale":"2024-03-26T21:22:09.374611+0000","last_undegraded":"2024-03-26T21:22:09.374611+0000","last_fullsized":"2024-03-26T21:22:09.374611+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:31:54.665171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374467+0000","last_change":"2024-03-26T21:21:31.216443+0000","last_active":"2024-03-26T21:22:09.374467+0000","last_peered":"2024-03-26T21:22:09.374467+0000","last_clean":"2024-03-26T21:22:09.374467+0000","last_became_active":"2024-03-26T21:21:31.215690+0000","last_became_peered":"2024-03-26T21:21:31.215690+0000","last_unstale":"2024-03-26T21:22:09.374467+0000","last_undegraded":"2024-03-26T21:22:09.374467+0000","last_fullsized":"2024-03-26T21:22:09.374467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:08:12.234978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377145+0000","last_change":"2024-03-26T21:21:29.206135+0000","last_active":"2024-03-26T21:22:09.377145+0000","last_peered":"2024-03-26T21:22:09.377145+0000","last_clean":"2024-03-26T21:22:09.377145+0000","last_became_active":"2024-03-26T21:21:29.205689+0000","last_became_peered":"2024-03-26T21:21:29.205689+0000","last_unstale":"2024-03-26T21:22:09.377145+0000","last_undegraded":"2024-03-26T21:22:09.377145+0000","last_fullsized":"2024-03-26T21:22:09.377145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:54:31.565734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378234+0000","last_change":"2024-03-26T21:21:25.193452+0000","last_active":"2024-03-26T21:22:09.378234+0000","last_peered":"2024-03-26T21:22:09.378234+0000","last_clean":"2024-03-26T21:22:09.378234+0000","last_became_active":"2024-03-26T21:21:25.193185+0000","last_became_peered":"2024-03-26T21:21:25.193185+0000","last_unstale":"2024-03-26T21:22:09.378234+0000","last_undegraded":"2024-03-26T21:22:09.378234+0000","last_fullsized":"2024-03-26T21:22:09.378234+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:34:43.864243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375855+0000","last_change":"2024-03-26T21:21:27.207383+0000","last_active":"2024-03-26T21:22:09.375855+0000","last_peered":"2024-03-26T21:22:09.375855+0000","last_clean":"2024-03-26T21:22:09.375855+0000","last_became_active":"2024-03-26T21:21:27.206804+0000","last_became_peered":"2024-03-26T21:21:27.206804+0000","last_unstale":"2024-03-26T21:22:09.375855+0000","last_undegraded":"2024-03-26T21:22:09.375855+0000","last_fullsized":"2024-03-26T21:22:09.375855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:18:08.358275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.493866+0000","last_change":"2024-03-26T21:21:31.217993+0000","last_active":"2024-03-26T21:22:09.493866+0000","last_peered":"2024-03-26T21:22:09.493866+0000","last_clean":"2024-03-26T21:22:09.493866+0000","last_became_active":"2024-03-26T21:21:31.217634+0000","last_became_peered":"2024-03-26T21:21:31.217634+0000","last_unstale":"2024-03-26T21:22:09.493866+0000","last_undegraded":"2024-03-26T21:22:09.493866+0000","last_fullsized":"2024-03-26T21:22:09.493866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:01:17.247200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375317+0000","last_change":"2024-03-26T21:21:29.205723+0000","last_active":"2024-03-26T21:22:09.375317+0000","last_peered":"2024-03-26T21:22:09.375317+0000","last_clean":"2024-03-26T21:22:09.375317+0000","last_became_active":"2024-03-26T21:21:29.205369+0000","last_became_peered":"2024-03-26T21:21:29.205369+0000","last_unstale":"2024-03-26T21:22:09.375317+0000","last_undegraded":"2024-03-26T21:22:09.375317+0000","last_fullsized":"2024-03-26T21:22:09.375317+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:55:36.680447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839773+0000","last_change":"2024-03-26T21:21:25.196056+0000","last_active":"2024-03-26T21:22:09.839773+0000","last_peered":"2024-03-26T21:22:09.839773+0000","last_clean":"2024-03-26T21:22:09.839773+0000","last_became_active":"2024-03-26T21:21:25.195560+0000","last_became_peered":"2024-03-26T21:21:25.195560+0000","last_unstale":"2024-03-26T21:22:09.839773+0000","last_undegraded":"2024-03-26T21:22:09.839773+0000","last_fullsized":"2024-03-26T21:22:09.839773+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:26:35.588884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838676+0000","last_change":"2024-03-26T21:21:48.182304+0000","last_active":"2024-03-26T21:22:09.838676+0000","last_peered":"2024-03-26T21:22:09.838676+0000","last_clean":"2024-03-26T21:22:09.838676+0000","last_became_active":"2024-03-26T21:21:48.174368+0000","last_became_peered":"2024-03-26T21:21:48.174368+0000","last_unstale":"2024-03-26T21:22:09.838676+0000","last_undegraded":"2024-03-26T21:22:09.838676+0000","last_fullsized":"2024-03-26T21:22:09.838676+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:39:58.801370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376591+0000","last_change":"2024-03-26T21:21:31.207576+0000","last_active":"2024-03-26T21:22:09.376591+0000","last_peered":"2024-03-26T21:22:09.376591+0000","last_clean":"2024-03-26T21:22:09.376591+0000","last_became_active":"2024-03-26T21:21:31.207258+0000","last_became_peered":"2024-03-26T21:21:31.207258+0000","last_unstale":"2024-03-26T21:22:09.376591+0000","last_undegraded":"2024-03-26T21:22:09.376591+0000","last_fullsized":"2024-03-26T21:22:09.376591+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:44:02.818399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249577+0000","last_change":"2024-03-26T21:21:29.198134+0000","last_active":"2024-03-26T21:22:45.249577+0000","last_peered":"2024-03-26T21:22:45.249577+0000","last_clean":"2024-03-26T21:22:45.249577+0000","last_became_active":"2024-03-26T21:21:29.197160+0000","last_became_peered":"2024-03-26T21:21:29.197160+0000","last_unstale":"2024-03-26T21:22:45.249577+0000","last_undegraded":"2024-03-26T21:22:45.249577+0000","last_fullsized":"2024-03-26T21:22:45.249577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:29:45.344203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494887+0000","last_change":"2024-03-26T21:21:25.190921+0000","last_active":"2024-03-26T21:22:09.494887+0000","last_peered":"2024-03-26T21:22:09.494887+0000","last_clean":"2024-03-26T21:22:09.494887+0000","last_became_active":"2024-03-26T21:21:25.190608+0000","last_became_peered":"2024-03-26T21:21:25.190608+0000","last_unstale":"2024-03-26T21:22:09.494887+0000","last_undegraded":"2024-03-26T21:22:09.494887+0000","last_fullsized":"2024-03-26T21:22:09.494887+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:37:24.149510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839601+0000","last_change":"2024-03-26T21:21:27.199318+0000","last_active":"2024-03-26T21:22:09.839601+0000","last_peered":"2024-03-26T21:22:09.839601+0000","last_clean":"2024-03-26T21:22:09.839601+0000","last_became_active":"2024-03-26T21:21:27.198803+0000","last_became_peered":"2024-03-26T21:21:27.198803+0000","last_unstale":"2024-03-26T21:22:09.839601+0000","last_undegraded":"2024-03-26T21:22:09.839601+0000","last_fullsized":"2024-03-26T21:22:09.839601+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:51:27.906494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.495061+0000","last_change":"2024-03-26T21:21:31.210765+0000","last_active":"2024-03-26T21:22:09.495061+0000","last_peered":"2024-03-26T21:22:09.495061+0000","last_clean":"2024-03-26T21:22:09.495061+0000","last_became_active":"2024-03-26T21:21:31.210414+0000","last_became_peered":"2024-03-26T21:21:31.210414+0000","last_unstale":"2024-03-26T21:22:09.495061+0000","last_undegraded":"2024-03-26T21:22:09.495061+0000","last_fullsized":"2024-03-26T21:22:09.495061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:08:13.712367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249891+0000","last_change":"2024-03-26T21:21:29.204650+0000","last_active":"2024-03-26T21:22:45.249891+0000","last_peered":"2024-03-26T21:22:45.249891+0000","last_clean":"2024-03-26T21:22:45.249891+0000","last_became_active":"2024-03-26T21:21:29.204354+0000","last_became_peered":"2024-03-26T21:21:29.204354+0000","last_unstale":"2024-03-26T21:22:45.249891+0000","last_undegraded":"2024-03-26T21:22:45.249891+0000","last_fullsized":"2024-03-26T21:22:45.249891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:49:36.163742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838959+0000","last_change":"2024-03-26T21:21:25.194063+0000","last_active":"2024-03-26T21:22:09.838959+0000","last_peered":"2024-03-26T21:22:09.838959+0000","last_clean":"2024-03-26T21:22:09.838959+0000","last_became_active":"2024-03-26T21:21:25.193176+0000","last_became_peered":"2024-03-26T21:21:25.193176+0000","last_unstale":"2024-03-26T21:22:09.838959+0000","last_undegraded":"2024-03-26T21:22:09.838959+0000","last_fullsized":"2024-03-26T21:22:09.838959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:56:46.803191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376643+0000","last_change":"2024-03-26T21:21:27.204286+0000","last_active":"2024-03-26T21:22:09.376643+0000","last_peered":"2024-03-26T21:22:09.376643+0000","last_clean":"2024-03-26T21:22:09.376643+0000","last_became_active":"2024-03-26T21:21:27.203710+0000","last_became_peered":"2024-03-26T21:21:27.203710+0000","last_unstale":"2024-03-26T21:22:09.376643+0000","last_undegraded":"2024-03-26T21:22:09.376643+0000","last_fullsized":"2024-03-26T21:22:09.376643+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:11:57.880443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840023+0000","last_change":"2024-03-26T21:21:31.348447+0000","last_active":"2024-03-26T21:22:09.840023+0000","last_peered":"2024-03-26T21:22:09.840023+0000","last_clean":"2024-03-26T21:22:09.840023+0000","last_became_active":"2024-03-26T21:21:31.348120+0000","last_became_peered":"2024-03-26T21:21:31.348120+0000","last_unstale":"2024-03-26T21:22:09.840023+0000","last_undegraded":"2024-03-26T21:22:09.840023+0000","last_fullsized":"2024-03-26T21:22:09.840023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:46:17.480158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839614+0000","last_change":"2024-03-26T21:21:29.205319+0000","last_active":"2024-03-26T21:22:09.839614+0000","last_peered":"2024-03-26T21:22:09.839614+0000","last_clean":"2024-03-26T21:22:09.839614+0000","last_became_active":"2024-03-26T21:21:29.205024+0000","last_became_peered":"2024-03-26T21:21:29.205024+0000","last_unstale":"2024-03-26T21:22:09.839614+0000","last_undegraded":"2024-03-26T21:22:09.839614+0000","last_fullsized":"2024-03-26T21:22:09.839614+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:07:59.496867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840638+0000","last_change":"2024-03-26T21:21:25.191540+0000","last_active":"2024-03-26T21:22:09.840638+0000","last_peered":"2024-03-26T21:22:09.840638+0000","last_clean":"2024-03-26T21:22:09.840638+0000","last_became_active":"2024-03-26T21:21:25.191291+0000","last_became_peered":"2024-03-26T21:21:25.191291+0000","last_unstale":"2024-03-26T21:22:09.840638+0000","last_undegraded":"2024-03-26T21:22:09.840638+0000","last_fullsized":"2024-03-26T21:22:09.840638+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:00:44.268906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376049+0000","last_change":"2024-03-26T21:21:27.204276+0000","last_active":"2024-03-26T21:22:09.376049+0000","last_peered":"2024-03-26T21:22:09.376049+0000","last_clean":"2024-03-26T21:22:09.376049+0000","last_became_active":"2024-03-26T21:21:27.203590+0000","last_became_peered":"2024-03-26T21:21:27.203590+0000","last_unstale":"2024-03-26T21:22:09.376049+0000","last_undegraded":"2024-03-26T21:22:09.376049+0000","last_fullsized":"2024-03-26T21:22:09.376049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:03:32.003907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838418+0000","last_change":"2024-03-26T21:21:31.212118+0000","last_active":"2024-03-26T21:22:09.838418+0000","last_peered":"2024-03-26T21:22:09.838418+0000","last_clean":"2024-03-26T21:22:09.838418+0000","last_became_active":"2024-03-26T21:21:31.211722+0000","last_became_peered":"2024-03-26T21:21:31.211722+0000","last_unstale":"2024-03-26T21:22:09.838418+0000","last_undegraded":"2024-03-26T21:22:09.838418+0000","last_fullsized":"2024-03-26T21:22:09.838418+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:36:10.162393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376254+0000","last_change":"2024-03-26T21:21:29.206711+0000","last_active":"2024-03-26T21:22:09.376254+0000","last_peered":"2024-03-26T21:22:09.376254+0000","last_clean":"2024-03-26T21:22:09.376254+0000","last_became_active":"2024-03-26T21:21:29.206404+0000","last_became_peered":"2024-03-26T21:21:29.206404+0000","last_unstale":"2024-03-26T21:22:09.376254+0000","last_undegraded":"2024-03-26T21:22:09.376254+0000","last_fullsized":"2024-03-26T21:22:09.376254+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:31:03.673863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838193+0000","last_change":"2024-03-26T21:21:25.189724+0000","last_active":"2024-03-26T21:22:09.838193+0000","last_peered":"2024-03-26T21:22:09.838193+0000","last_clean":"2024-03-26T21:22:09.838193+0000","last_became_active":"2024-03-26T21:21:25.189454+0000","last_became_peered":"2024-03-26T21:21:25.189454+0000","last_unstale":"2024-03-26T21:22:09.838193+0000","last_undegraded":"2024-03-26T21:22:09.838193+0000","last_fullsized":"2024-03-26T21:22:09.838193+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:09:42.669740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494684+0000","last_change":"2024-03-26T21:21:27.199475+0000","last_active":"2024-03-26T21:22:09.494684+0000","last_peered":"2024-03-26T21:22:09.494684+0000","last_clean":"2024-03-26T21:22:09.494684+0000","last_became_active":"2024-03-26T21:21:27.199144+0000","last_became_peered":"2024-03-26T21:21:27.199144+0000","last_unstale":"2024-03-26T21:22:09.494684+0000","last_undegraded":"2024-03-26T21:22:09.494684+0000","last_fullsized":"2024-03-26T21:22:09.494684+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:46:09.380308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375399+0000","last_change":"2024-03-26T21:21:31.219944+0000","last_active":"2024-03-26T21:22:09.375399+0000","last_peered":"2024-03-26T21:22:09.375399+0000","last_clean":"2024-03-26T21:22:09.375399+0000","last_became_active":"2024-03-26T21:21:31.219577+0000","last_became_peered":"2024-03-26T21:21:31.219577+0000","last_unstale":"2024-03-26T21:22:09.375399+0000","last_undegraded":"2024-03-26T21:22:09.375399+0000","last_fullsized":"2024-03-26T21:22:09.375399+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:22:57.749637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376315+0000","last_change":"2024-03-26T21:21:25.188671+0000","last_active":"2024-03-26T21:22:09.376315+0000","last_peered":"2024-03-26T21:22:09.376315+0000","last_clean":"2024-03-26T21:22:09.376315+0000","last_became_active":"2024-03-26T21:21:25.187132+0000","last_became_peered":"2024-03-26T21:21:25.187132+0000","last_unstale":"2024-03-26T21:22:09.376315+0000","last_undegraded":"2024-03-26T21:22:09.376315+0000","last_fullsized":"2024-03-26T21:22:09.376315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:07:55.551378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375609+0000","last_change":"2024-03-26T21:21:27.208795+0000","last_active":"2024-03-26T21:22:09.375609+0000","last_peered":"2024-03-26T21:22:09.375609+0000","last_clean":"2024-03-26T21:22:09.375609+0000","last_became_active":"2024-03-26T21:21:27.208450+0000","last_became_peered":"2024-03-26T21:21:27.208450+0000","last_unstale":"2024-03-26T21:22:09.375609+0000","last_undegraded":"2024-03-26T21:22:09.375609+0000","last_fullsized":"2024-03-26T21:22:09.375609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:24:43.770586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838190+0000","last_change":"2024-03-26T21:21:29.209719+0000","last_active":"2024-03-26T21:22:09.838190+0000","last_peered":"2024-03-26T21:22:09.838190+0000","last_clean":"2024-03-26T21:22:09.838190+0000","last_became_active":"2024-03-26T21:21:29.209351+0000","last_became_peered":"2024-03-26T21:21:29.209351+0000","last_unstale":"2024-03-26T21:22:09.838190+0000","last_undegraded":"2024-03-26T21:22:09.838190+0000","last_fullsized":"2024-03-26T21:22:09.838190+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:30:58.632440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.493883+0000","last_change":"2024-03-26T21:21:31.210011+0000","last_active":"2024-03-26T21:22:09.493883+0000","last_peered":"2024-03-26T21:22:09.493883+0000","last_clean":"2024-03-26T21:22:09.493883+0000","last_became_active":"2024-03-26T21:21:31.209693+0000","last_became_peered":"2024-03-26T21:21:31.209693+0000","last_unstale":"2024-03-26T21:22:09.493883+0000","last_undegraded":"2024-03-26T21:22:09.493883+0000","last_fullsized":"2024-03-26T21:22:09.493883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:52:13.737074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377796+0000","last_change":"2024-03-26T21:21:25.188069+0000","last_active":"2024-03-26T21:22:09.377796+0000","last_peered":"2024-03-26T21:22:09.377796+0000","last_clean":"2024-03-26T21:22:09.377796+0000","last_became_active":"2024-03-26T21:21:25.187545+0000","last_became_peered":"2024-03-26T21:21:25.187545+0000","last_unstale":"2024-03-26T21:22:09.377796+0000","last_undegraded":"2024-03-26T21:22:09.377796+0000","last_fullsized":"2024-03-26T21:22:09.377796+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:45:58.086524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377019+0000","last_change":"2024-03-26T21:21:27.204561+0000","last_active":"2024-03-26T21:22:09.377019+0000","last_peered":"2024-03-26T21:22:09.377019+0000","last_clean":"2024-03-26T21:22:09.377019+0000","last_became_active":"2024-03-26T21:21:27.203458+0000","last_became_peered":"2024-03-26T21:21:27.203458+0000","last_unstale":"2024-03-26T21:22:09.377019+0000","last_undegraded":"2024-03-26T21:22:09.377019+0000","last_fullsized":"2024-03-26T21:22:09.377019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:45:59.929399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249573+0000","last_change":"2024-03-26T21:21:29.199024+0000","last_active":"2024-03-26T21:22:45.249573+0000","last_peered":"2024-03-26T21:22:45.249573+0000","last_clean":"2024-03-26T21:22:45.249573+0000","last_became_active":"2024-03-26T21:21:29.198694+0000","last_became_peered":"2024-03-26T21:21:29.198694+0000","last_unstale":"2024-03-26T21:22:45.249573+0000","last_undegraded":"2024-03-26T21:22:45.249573+0000","last_fullsized":"2024-03-26T21:22:45.249573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:44:20.562098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375771+0000","last_change":"2024-03-26T21:21:31.213146+0000","last_active":"2024-03-26T21:22:09.375771+0000","last_peered":"2024-03-26T21:22:09.375771+0000","last_clean":"2024-03-26T21:22:09.375771+0000","last_became_active":"2024-03-26T21:21:31.212791+0000","last_became_peered":"2024-03-26T21:21:31.212791+0000","last_unstale":"2024-03-26T21:22:09.375771+0000","last_undegraded":"2024-03-26T21:22:09.375771+0000","last_fullsized":"2024-03-26T21:22:09.375771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:10:23.445388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839060+0000","last_change":"2024-03-26T21:21:25.195053+0000","last_active":"2024-03-26T21:22:09.839060+0000","last_peered":"2024-03-26T21:22:09.839060+0000","last_clean":"2024-03-26T21:22:09.839060+0000","last_became_active":"2024-03-26T21:21:25.194460+0000","last_became_peered":"2024-03-26T21:21:25.194460+0000","last_unstale":"2024-03-26T21:22:09.839060+0000","last_undegraded":"2024-03-26T21:22:09.839060+0000","last_fullsized":"2024-03-26T21:22:09.839060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:31:47.477889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":95,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249699+0000","last_change":"2024-03-26T21:21:27.199020+0000","last_active":"2024-03-26T21:22:45.249699+0000","last_peered":"2024-03-26T21:22:45.249699+0000","last_clean":"2024-03-26T21:22:45.249699+0000","last_became_active":"2024-03-26T21:21:27.198722+0000","last_became_peered":"2024-03-26T21:21:27.198722+0000","last_unstale":"2024-03-26T21:22:45.249699+0000","last_undegraded":"2024-03-26T21:22:45.249699+0000","last_fullsized":"2024-03-26T21:22:45.249699+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:54:48.210382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494822+0000","last_change":"2024-03-26T21:21:29.205845+0000","last_active":"2024-03-26T21:22:09.494822+0000","last_peered":"2024-03-26T21:22:09.494822+0000","last_clean":"2024-03-26T21:22:09.494822+0000","last_became_active":"2024-03-26T21:21:29.205332+0000","last_became_peered":"2024-03-26T21:21:29.205332+0000","last_unstale":"2024-03-26T21:22:09.494822+0000","last_undegraded":"2024-03-26T21:22:09.494822+0000","last_fullsized":"2024-03-26T21:22:09.494822+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:32:49.582713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840457+0000","last_change":"2024-03-26T21:21:31.349116+0000","last_active":"2024-03-26T21:22:09.840457+0000","last_peered":"2024-03-26T21:22:09.840457+0000","last_clean":"2024-03-26T21:22:09.840457+0000","last_became_active":"2024-03-26T21:21:31.348805+0000","last_became_peered":"2024-03-26T21:21:31.348805+0000","last_unstale":"2024-03-26T21:22:09.840457+0000","last_undegraded":"2024-03-26T21:22:09.840457+0000","last_fullsized":"2024-03-26T21:22:09.840457+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:30:25.112916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'5","reported_seq":122,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:49.069982+0000","last_change":"2024-03-26T21:21:29.205538+0000","last_active":"2024-03-26T21:22:49.069982+0000","last_peered":"2024-03-26T21:22:49.069982+0000","last_clean":"2024-03-26T21:22:49.069982+0000","last_became_active":"2024-03-26T21:21:25.188351+0000","last_became_peered":"2024-03-26T21:21:25.188351+0000","last_unstale":"2024-03-26T21:22:49.069982+0000","last_undegraded":"2024-03-26T21:22:49.069982+0000","last_fullsized":"2024-03-26T21:22:49.069982+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:21:34.467438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00100063,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838161+0000","last_change":"2024-03-26T21:21:25.184917+0000","last_active":"2024-03-26T21:22:09.838161+0000","last_peered":"2024-03-26T21:22:09.838161+0000","last_clean":"2024-03-26T21:22:09.838161+0000","last_became_active":"2024-03-26T21:21:25.184619+0000","last_became_peered":"2024-03-26T21:21:25.184619+0000","last_unstale":"2024-03-26T21:22:09.838161+0000","last_undegraded":"2024-03-26T21:22:09.838161+0000","last_fullsized":"2024-03-26T21:22:09.838161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:26:03.908836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.379149+0000","last_change":"2024-03-26T21:21:27.195400+0000","last_active":"2024-03-26T21:22:09.379149+0000","last_peered":"2024-03-26T21:22:09.379149+0000","last_clean":"2024-03-26T21:22:09.379149+0000","last_became_active":"2024-03-26T21:21:27.195082+0000","last_became_peered":"2024-03-26T21:21:27.195082+0000","last_unstale":"2024-03-26T21:22:09.379149+0000","last_undegraded":"2024-03-26T21:22:09.379149+0000","last_fullsized":"2024-03-26T21:22:09.379149+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:40:57.955259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838613+0000","last_change":"2024-03-26T21:21:29.204082+0000","last_active":"2024-03-26T21:22:09.838613+0000","last_peered":"2024-03-26T21:22:09.838613+0000","last_clean":"2024-03-26T21:22:09.838613+0000","last_became_active":"2024-03-26T21:21:29.203741+0000","last_became_peered":"2024-03-26T21:21:29.203741+0000","last_unstale":"2024-03-26T21:22:09.838613+0000","last_undegraded":"2024-03-26T21:22:09.838613+0000","last_fullsized":"2024-03-26T21:22:09.838613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T03:15:20.663676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840378+0000","last_change":"2024-03-26T21:21:31.214988+0000","last_active":"2024-03-26T21:22:09.840378+0000","last_peered":"2024-03-26T21:22:09.840378+0000","last_clean":"2024-03-26T21:22:09.840378+0000","last_became_active":"2024-03-26T21:21:31.214654+0000","last_became_peered":"2024-03-26T21:21:31.214654+0000","last_unstale":"2024-03-26T21:22:09.840378+0000","last_undegraded":"2024-03-26T21:22:09.840378+0000","last_fullsized":"2024-03-26T21:22:09.840378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:48:12.886918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375028+0000","last_change":"2024-03-26T21:21:29.194135+0000","last_active":"2024-03-26T21:22:09.375028+0000","last_peered":"2024-03-26T21:22:09.375028+0000","last_clean":"2024-03-26T21:22:09.375028+0000","last_became_active":"2024-03-26T21:21:25.193663+0000","last_became_peered":"2024-03-26T21:21:25.193663+0000","last_unstale":"2024-03-26T21:22:09.375028+0000","last_undegraded":"2024-03-26T21:22:09.375028+0000","last_fullsized":"2024-03-26T21:22:09.375028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:40:57.855172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002579769,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838769+0000","last_change":"2024-03-26T21:21:48.177755+0000","last_active":"2024-03-26T21:22:09.838769+0000","last_peered":"2024-03-26T21:22:09.838769+0000","last_clean":"2024-03-26T21:22:09.838769+0000","last_became_active":"2024-03-26T21:21:48.177292+0000","last_became_peered":"2024-03-26T21:21:48.177292+0000","last_unstale":"2024-03-26T21:22:09.838769+0000","last_undegraded":"2024-03-26T21:22:09.838769+0000","last_fullsized":"2024-03-26T21:22:09.838769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:08:22.638059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'128","reported_seq":272,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:11.473088+0000","last_change":"2024-03-26T21:21:14.714857+0000","last_active":"2024-03-26T21:22:11.473088+0000","last_peered":"2024-03-26T21:22:11.473088+0000","last_clean":"2024-03-26T21:22:11.473088+0000","last_became_active":"2024-03-26T21:21:14.714386+0000","last_became_peered":"2024-03-26T21:21:14.714386+0000","last_unstale":"2024-03-26T21:22:11.473088+0000","last_undegraded":"2024-03-26T21:22:11.473088+0000","last_fullsized":"2024-03-26T21:22:11.473088+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:19:03.226555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:19:03.226555+0000","last_clean_scrub_stamp":"2024-03-26T21:19:03.226555+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:23:38.758836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249938+0000","last_change":"2024-03-26T21:21:27.197740+0000","last_active":"2024-03-26T21:22:45.249938+0000","last_peered":"2024-03-26T21:22:45.249938+0000","last_clean":"2024-03-26T21:22:45.249938+0000","last_became_active":"2024-03-26T21:21:27.197389+0000","last_became_peered":"2024-03-26T21:21:27.197389+0000","last_unstale":"2024-03-26T21:22:45.249938+0000","last_undegraded":"2024-03-26T21:22:45.249938+0000","last_fullsized":"2024-03-26T21:22:45.249938+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:20:02.610847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840003+0000","last_change":"2024-03-26T21:21:29.199860+0000","last_active":"2024-03-26T21:22:09.840003+0000","last_peered":"2024-03-26T21:22:09.840003+0000","last_clean":"2024-03-26T21:22:09.840003+0000","last_became_active":"2024-03-26T21:21:29.199549+0000","last_became_peered":"2024-03-26T21:21:29.199549+0000","last_unstale":"2024-03-26T21:22:09.840003+0000","last_undegraded":"2024-03-26T21:22:09.840003+0000","last_fullsized":"2024-03-26T21:22:09.840003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:13:43.893929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377566+0000","last_change":"2024-03-26T21:21:31.218936+0000","last_active":"2024-03-26T21:22:09.377566+0000","last_peered":"2024-03-26T21:22:09.377566+0000","last_clean":"2024-03-26T21:22:09.377566+0000","last_became_active":"2024-03-26T21:21:31.218620+0000","last_became_peered":"2024-03-26T21:21:31.218620+0000","last_unstale":"2024-03-26T21:22:09.377566+0000","last_undegraded":"2024-03-26T21:22:09.377566+0000","last_fullsized":"2024-03-26T21:22:09.377566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:27:44.780741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494329+0000","last_change":"2024-03-26T21:21:29.195555+0000","last_active":"2024-03-26T21:22:09.494329+0000","last_peered":"2024-03-26T21:22:09.494329+0000","last_clean":"2024-03-26T21:22:09.494329+0000","last_became_active":"2024-03-26T21:21:25.192273+0000","last_became_peered":"2024-03-26T21:21:25.192273+0000","last_unstale":"2024-03-26T21:22:09.494329+0000","last_undegraded":"2024-03-26T21:22:09.494329+0000","last_fullsized":"2024-03-26T21:22:09.494329+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:59:22.247736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039106999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839177+0000","last_change":"2024-03-26T21:21:25.195379+0000","last_active":"2024-03-26T21:22:09.839177+0000","last_peered":"2024-03-26T21:22:09.839177+0000","last_clean":"2024-03-26T21:22:09.839177+0000","last_became_active":"2024-03-26T21:21:25.195119+0000","last_became_peered":"2024-03-26T21:21:25.195119+0000","last_unstale":"2024-03-26T21:22:09.839177+0000","last_undegraded":"2024-03-26T21:22:09.839177+0000","last_fullsized":"2024-03-26T21:22:09.839177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:44:22.259071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494545+0000","last_change":"2024-03-26T21:21:48.188101+0000","last_active":"2024-03-26T21:22:09.494545+0000","last_peered":"2024-03-26T21:22:09.494545+0000","last_clean":"2024-03-26T21:22:09.494545+0000","last_became_active":"2024-03-26T21:21:48.177016+0000","last_became_peered":"2024-03-26T21:21:48.177016+0000","last_unstale":"2024-03-26T21:22:09.494545+0000","last_undegraded":"2024-03-26T21:22:09.494545+0000","last_fullsized":"2024-03-26T21:22:09.494545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:59:47.697063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840289+0000","last_change":"2024-03-26T21:21:29.208167+0000","last_active":"2024-03-26T21:22:09.840289+0000","last_peered":"2024-03-26T21:22:09.840289+0000","last_clean":"2024-03-26T21:22:09.840289+0000","last_became_active":"2024-03-26T21:21:29.207799+0000","last_became_peered":"2024-03-26T21:21:29.207799+0000","last_unstale":"2024-03-26T21:22:09.840289+0000","last_undegraded":"2024-03-26T21:22:09.840289+0000","last_fullsized":"2024-03-26T21:22:09.840289+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:29:56.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376278+0000","last_change":"2024-03-26T21:21:31.348097+0000","last_active":"2024-03-26T21:22:09.376278+0000","last_peered":"2024-03-26T21:22:09.376278+0000","last_clean":"2024-03-26T21:22:09.376278+0000","last_became_active":"2024-03-26T21:21:31.347734+0000","last_became_peered":"2024-03-26T21:21:31.347734+0000","last_unstale":"2024-03-26T21:22:09.376278+0000","last_undegraded":"2024-03-26T21:22:09.376278+0000","last_fullsized":"2024-03-26T21:22:09.376278+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:46:26.345041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840493+0000","last_change":"2024-03-26T21:21:25.190385+0000","last_active":"2024-03-26T21:22:09.840493+0000","last_peered":"2024-03-26T21:22:09.840493+0000","last_clean":"2024-03-26T21:22:09.840493+0000","last_became_active":"2024-03-26T21:21:25.189649+0000","last_became_peered":"2024-03-26T21:21:25.189649+0000","last_unstale":"2024-03-26T21:22:09.840493+0000","last_undegraded":"2024-03-26T21:22:09.840493+0000","last_fullsized":"2024-03-26T21:22:09.840493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:19:57.794225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494757+0000","last_change":"2024-03-26T21:21:27.193764+0000","last_active":"2024-03-26T21:22:09.494757+0000","last_peered":"2024-03-26T21:22:09.494757+0000","last_clean":"2024-03-26T21:22:09.494757+0000","last_became_active":"2024-03-26T21:21:27.193424+0000","last_became_peered":"2024-03-26T21:21:27.193424+0000","last_unstale":"2024-03-26T21:22:09.494757+0000","last_undegraded":"2024-03-26T21:22:09.494757+0000","last_fullsized":"2024-03-26T21:22:09.494757+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:46:44.280198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839267+0000","last_change":"2024-03-26T21:21:29.198259+0000","last_active":"2024-03-26T21:22:09.839267+0000","last_peered":"2024-03-26T21:22:09.839267+0000","last_clean":"2024-03-26T21:22:09.839267+0000","last_became_active":"2024-03-26T21:21:29.197988+0000","last_became_peered":"2024-03-26T21:21:29.197988+0000","last_unstale":"2024-03-26T21:22:09.839267+0000","last_undegraded":"2024-03-26T21:22:09.839267+0000","last_fullsized":"2024-03-26T21:22:09.839267+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:56:53.580647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375976+0000","last_change":"2024-03-26T21:21:31.209675+0000","last_active":"2024-03-26T21:22:09.375976+0000","last_peered":"2024-03-26T21:22:09.375976+0000","last_clean":"2024-03-26T21:22:09.375976+0000","last_became_active":"2024-03-26T21:21:31.209349+0000","last_became_peered":"2024-03-26T21:21:31.209349+0000","last_unstale":"2024-03-26T21:22:09.375976+0000","last_undegraded":"2024-03-26T21:22:09.375976+0000","last_fullsized":"2024-03-26T21:22:09.375976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:02:35.862171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378837+0000","last_change":"2024-03-26T21:21:25.194315+0000","last_active":"2024-03-26T21:22:09.378837+0000","last_peered":"2024-03-26T21:22:09.378837+0000","last_clean":"2024-03-26T21:22:09.378837+0000","last_became_active":"2024-03-26T21:21:25.193875+0000","last_became_peered":"2024-03-26T21:21:25.193875+0000","last_unstale":"2024-03-26T21:22:09.378837+0000","last_undegraded":"2024-03-26T21:22:09.378837+0000","last_fullsized":"2024-03-26T21:22:09.378837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:21:43.442879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.374492+0000","last_change":"2024-03-26T21:21:27.204833+0000","last_active":"2024-03-26T21:22:09.374492+0000","last_peered":"2024-03-26T21:22:09.374492+0000","last_clean":"2024-03-26T21:22:09.374492+0000","last_became_active":"2024-03-26T21:21:27.204524+0000","last_became_peered":"2024-03-26T21:21:27.204524+0000","last_unstale":"2024-03-26T21:22:09.374492+0000","last_undegraded":"2024-03-26T21:22:09.374492+0000","last_fullsized":"2024-03-26T21:22:09.374492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:09:56.359121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375749+0000","last_change":"2024-03-26T21:21:29.198837+0000","last_active":"2024-03-26T21:22:09.375749+0000","last_peered":"2024-03-26T21:22:09.375749+0000","last_clean":"2024-03-26T21:22:09.375749+0000","last_became_active":"2024-03-26T21:21:29.198142+0000","last_became_peered":"2024-03-26T21:21:29.198142+0000","last_unstale":"2024-03-26T21:22:09.375749+0000","last_undegraded":"2024-03-26T21:22:09.375749+0000","last_fullsized":"2024-03-26T21:22:09.375749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:41:27.234335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840939+0000","last_change":"2024-03-26T21:21:31.348461+0000","last_active":"2024-03-26T21:22:09.840939+0000","last_peered":"2024-03-26T21:22:09.840939+0000","last_clean":"2024-03-26T21:22:09.840939+0000","last_became_active":"2024-03-26T21:21:31.348127+0000","last_became_peered":"2024-03-26T21:21:31.348127+0000","last_unstale":"2024-03-26T21:22:09.840939+0000","last_undegraded":"2024-03-26T21:22:09.840939+0000","last_fullsized":"2024-03-26T21:22:09.840939+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:15:13.159606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840827+0000","last_change":"2024-03-26T21:21:25.188837+0000","last_active":"2024-03-26T21:22:09.840827+0000","last_peered":"2024-03-26T21:22:09.840827+0000","last_clean":"2024-03-26T21:22:09.840827+0000","last_became_active":"2024-03-26T21:21:25.188536+0000","last_became_peered":"2024-03-26T21:21:25.188536+0000","last_unstale":"2024-03-26T21:22:09.840827+0000","last_undegraded":"2024-03-26T21:22:09.840827+0000","last_fullsized":"2024-03-26T21:22:09.840827+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:30:45.797186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375647+0000","last_change":"2024-03-26T21:21:27.199658+0000","last_active":"2024-03-26T21:22:09.375647+0000","last_peered":"2024-03-26T21:22:09.375647+0000","last_clean":"2024-03-26T21:22:09.375647+0000","last_became_active":"2024-03-26T21:21:27.199336+0000","last_became_peered":"2024-03-26T21:21:27.199336+0000","last_unstale":"2024-03-26T21:22:09.375647+0000","last_undegraded":"2024-03-26T21:22:09.375647+0000","last_fullsized":"2024-03-26T21:22:09.375647+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:15:41.049691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249569+0000","last_change":"2024-03-26T21:21:29.209346+0000","last_active":"2024-03-26T21:22:45.249569+0000","last_peered":"2024-03-26T21:22:45.249569+0000","last_clean":"2024-03-26T21:22:45.249569+0000","last_became_active":"2024-03-26T21:21:29.209004+0000","last_became_peered":"2024-03-26T21:21:29.209004+0000","last_unstale":"2024-03-26T21:22:45.249569+0000","last_undegraded":"2024-03-26T21:22:45.249569+0000","last_fullsized":"2024-03-26T21:22:45.249569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:22:33.834366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376493+0000","last_change":"2024-03-26T21:21:31.347339+0000","last_active":"2024-03-26T21:22:09.376493+0000","last_peered":"2024-03-26T21:22:09.376493+0000","last_clean":"2024-03-26T21:22:09.376493+0000","last_became_active":"2024-03-26T21:21:31.346912+0000","last_became_peered":"2024-03-26T21:21:31.346912+0000","last_unstale":"2024-03-26T21:22:09.376493+0000","last_undegraded":"2024-03-26T21:22:09.376493+0000","last_fullsized":"2024-03-26T21:22:09.376493+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:12:27.250572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.840136+0000","last_change":"2024-03-26T21:21:29.198185+0000","last_active":"2024-03-26T21:22:09.840136+0000","last_peered":"2024-03-26T21:22:09.840136+0000","last_clean":"2024-03-26T21:22:09.840136+0000","last_became_active":"2024-03-26T21:21:29.197858+0000","last_became_peered":"2024-03-26T21:21:29.197858+0000","last_unstale":"2024-03-26T21:22:09.840136+0000","last_undegraded":"2024-03-26T21:22:09.840136+0000","last_fullsized":"2024-03-26T21:22:09.840136+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:10:29.651320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839375+0000","last_change":"2024-03-26T21:21:25.188797+0000","last_active":"2024-03-26T21:22:09.839375+0000","last_peered":"2024-03-26T21:22:09.839375+0000","last_clean":"2024-03-26T21:22:09.839375+0000","last_became_active":"2024-03-26T21:21:25.188492+0000","last_became_peered":"2024-03-26T21:21:25.188492+0000","last_unstale":"2024-03-26T21:22:09.839375+0000","last_undegraded":"2024-03-26T21:22:09.839375+0000","last_fullsized":"2024-03-26T21:22:09.839375+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T21:37:47.176328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376625+0000","last_change":"2024-03-26T21:21:27.206582+0000","last_active":"2024-03-26T21:22:09.376625+0000","last_peered":"2024-03-26T21:22:09.376625+0000","last_clean":"2024-03-26T21:22:09.376625+0000","last_became_active":"2024-03-26T21:21:27.206272+0000","last_became_peered":"2024-03-26T21:21:27.206272+0000","last_unstale":"2024-03-26T21:22:09.376625+0000","last_undegraded":"2024-03-26T21:22:09.376625+0000","last_fullsized":"2024-03-26T21:22:09.376625+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:19:03.157404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839290+0000","last_change":"2024-03-26T21:21:31.219142+0000","last_active":"2024-03-26T21:22:09.839290+0000","last_peered":"2024-03-26T21:22:09.839290+0000","last_clean":"2024-03-26T21:22:09.839290+0000","last_became_active":"2024-03-26T21:21:31.218915+0000","last_became_peered":"2024-03-26T21:21:31.218915+0000","last_unstale":"2024-03-26T21:22:09.839290+0000","last_undegraded":"2024-03-26T21:22:09.839290+0000","last_fullsized":"2024-03-26T21:22:09.839290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:12:30.083392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249895+0000","last_change":"2024-03-26T21:21:29.197672+0000","last_active":"2024-03-26T21:22:45.249895+0000","last_peered":"2024-03-26T21:22:45.249895+0000","last_clean":"2024-03-26T21:22:45.249895+0000","last_became_active":"2024-03-26T21:21:29.197295+0000","last_became_peered":"2024-03-26T21:21:29.197295+0000","last_unstale":"2024-03-26T21:22:45.249895+0000","last_undegraded":"2024-03-26T21:22:45.249895+0000","last_fullsized":"2024-03-26T21:22:45.249895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:03:16.385162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376525+0000","last_change":"2024-03-26T21:21:25.193057+0000","last_active":"2024-03-26T21:22:09.376525+0000","last_peered":"2024-03-26T21:22:09.376525+0000","last_clean":"2024-03-26T21:22:09.376525+0000","last_became_active":"2024-03-26T21:21:25.192721+0000","last_became_peered":"2024-03-26T21:21:25.192721+0000","last_unstale":"2024-03-26T21:22:09.376525+0000","last_undegraded":"2024-03-26T21:22:09.376525+0000","last_fullsized":"2024-03-26T21:22:09.376525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:18:29.762531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.379273+0000","last_change":"2024-03-26T21:21:27.204279+0000","last_active":"2024-03-26T21:22:09.379273+0000","last_peered":"2024-03-26T21:22:09.379273+0000","last_clean":"2024-03-26T21:22:09.379273+0000","last_became_active":"2024-03-26T21:21:27.203993+0000","last_became_peered":"2024-03-26T21:21:27.203993+0000","last_unstale":"2024-03-26T21:22:09.379273+0000","last_undegraded":"2024-03-26T21:22:09.379273+0000","last_fullsized":"2024-03-26T21:22:09.379273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T01:11:44.363251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376343+0000","last_change":"2024-03-26T21:21:31.216339+0000","last_active":"2024-03-26T21:22:09.376343+0000","last_peered":"2024-03-26T21:22:09.376343+0000","last_clean":"2024-03-26T21:22:09.376343+0000","last_became_active":"2024-03-26T21:21:31.216056+0000","last_became_peered":"2024-03-26T21:21:31.216056+0000","last_unstale":"2024-03-26T21:22:09.376343+0000","last_undegraded":"2024-03-26T21:22:09.376343+0000","last_fullsized":"2024-03-26T21:22:09.376343+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:55:36.596444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249582+0000","last_change":"2024-03-26T21:21:29.202671+0000","last_active":"2024-03-26T21:22:45.249582+0000","last_peered":"2024-03-26T21:22:45.249582+0000","last_clean":"2024-03-26T21:22:45.249582+0000","last_became_active":"2024-03-26T21:21:29.202282+0000","last_became_peered":"2024-03-26T21:21:29.202282+0000","last_unstale":"2024-03-26T21:22:45.249582+0000","last_undegraded":"2024-03-26T21:22:45.249582+0000","last_fullsized":"2024-03-26T21:22:45.249582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:37:25.062725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376920+0000","last_change":"2024-03-26T21:21:25.194419+0000","last_active":"2024-03-26T21:22:09.376920+0000","last_peered":"2024-03-26T21:22:09.376920+0000","last_clean":"2024-03-26T21:22:09.376920+0000","last_became_active":"2024-03-26T21:21:25.194067+0000","last_became_peered":"2024-03-26T21:21:25.194067+0000","last_unstale":"2024-03-26T21:22:09.376920+0000","last_undegraded":"2024-03-26T21:22:09.376920+0000","last_fullsized":"2024-03-26T21:22:09.376920+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T09:15:27.805137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.375812+0000","last_change":"2024-03-26T21:21:27.207639+0000","last_active":"2024-03-26T21:22:09.375812+0000","last_peered":"2024-03-26T21:22:09.375812+0000","last_clean":"2024-03-26T21:22:09.375812+0000","last_became_active":"2024-03-26T21:21:27.207347+0000","last_became_peered":"2024-03-26T21:21:27.207347+0000","last_unstale":"2024-03-26T21:22:09.375812+0000","last_undegraded":"2024-03-26T21:22:09.375812+0000","last_fullsized":"2024-03-26T21:22:09.375812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T22:27:40.075866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838792+0000","last_change":"2024-03-26T21:21:31.217737+0000","last_active":"2024-03-26T21:22:09.838792+0000","last_peered":"2024-03-26T21:22:09.838792+0000","last_clean":"2024-03-26T21:22:09.838792+0000","last_became_active":"2024-03-26T21:21:31.217407+0000","last_became_peered":"2024-03-26T21:21:31.217407+0000","last_unstale":"2024-03-26T21:22:09.838792+0000","last_undegraded":"2024-03-26T21:22:09.838792+0000","last_fullsized":"2024-03-26T21:22:09.838792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:04:58.041327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.377656+0000","last_change":"2024-03-26T21:21:29.206774+0000","last_active":"2024-03-26T21:22:09.377656+0000","last_peered":"2024-03-26T21:22:09.377656+0000","last_clean":"2024-03-26T21:22:09.377656+0000","last_became_active":"2024-03-26T21:21:29.206472+0000","last_became_peered":"2024-03-26T21:21:29.206472+0000","last_unstale":"2024-03-26T21:22:09.377656+0000","last_undegraded":"2024-03-26T21:22:09.377656+0000","last_fullsized":"2024-03-26T21:22:09.377656+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:52:43.335726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839303+0000","last_change":"2024-03-26T21:21:25.194241+0000","last_active":"2024-03-26T21:22:09.839303+0000","last_peered":"2024-03-26T21:22:09.839303+0000","last_clean":"2024-03-26T21:22:09.839303+0000","last_became_active":"2024-03-26T21:21:25.193964+0000","last_became_peered":"2024-03-26T21:21:25.193964+0000","last_unstale":"2024-03-26T21:22:09.839303+0000","last_undegraded":"2024-03-26T21:22:09.839303+0000","last_fullsized":"2024-03-26T21:22:09.839303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T07:55:33.520719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.376132+0000","last_change":"2024-03-26T21:21:27.206502+0000","last_active":"2024-03-26T21:22:09.376132+0000","last_peered":"2024-03-26T21:22:09.376132+0000","last_clean":"2024-03-26T21:22:09.376132+0000","last_became_active":"2024-03-26T21:21:27.206173+0000","last_became_peered":"2024-03-26T21:21:27.206173+0000","last_unstale":"2024-03-26T21:22:09.376132+0000","last_undegraded":"2024-03-26T21:22:09.376132+0000","last_fullsized":"2024-03-26T21:22:09.376132+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:46:16.365138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839066+0000","last_change":"2024-03-26T21:21:31.209384+0000","last_active":"2024-03-26T21:22:09.839066+0000","last_peered":"2024-03-26T21:22:09.839066+0000","last_clean":"2024-03-26T21:22:09.839066+0000","last_became_active":"2024-03-26T21:21:31.209071+0000","last_became_peered":"2024-03-26T21:21:31.209071+0000","last_unstale":"2024-03-26T21:22:09.839066+0000","last_undegraded":"2024-03-26T21:22:09.839066+0000","last_fullsized":"2024-03-26T21:22:09.839066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:56:14.900945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838498+0000","last_change":"2024-03-26T21:21:29.200667+0000","last_active":"2024-03-26T21:22:09.838498+0000","last_peered":"2024-03-26T21:22:09.838498+0000","last_clean":"2024-03-26T21:22:09.838498+0000","last_became_active":"2024-03-26T21:21:29.200345+0000","last_became_peered":"2024-03-26T21:21:29.200345+0000","last_unstale":"2024-03-26T21:22:09.838498+0000","last_undegraded":"2024-03-26T21:22:09.838498+0000","last_fullsized":"2024-03-26T21:22:09.838498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:20:43.952201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838575+0000","last_change":"2024-03-26T21:21:25.188968+0000","last_active":"2024-03-26T21:22:09.838575+0000","last_peered":"2024-03-26T21:22:09.838575+0000","last_clean":"2024-03-26T21:22:09.838575+0000","last_became_active":"2024-03-26T21:21:25.188613+0000","last_became_peered":"2024-03-26T21:21:25.188613+0000","last_unstale":"2024-03-26T21:22:09.838575+0000","last_undegraded":"2024-03-26T21:22:09.838575+0000","last_fullsized":"2024-03-26T21:22:09.838575+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T05:01:31.748061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.839029+0000","last_change":"2024-03-26T21:21:27.196031+0000","last_active":"2024-03-26T21:22:09.839029+0000","last_peered":"2024-03-26T21:22:09.839029+0000","last_clean":"2024-03-26T21:22:09.839029+0000","last_became_active":"2024-03-26T21:21:27.195718+0000","last_became_peered":"2024-03-26T21:21:27.195718+0000","last_unstale":"2024-03-26T21:22:09.839029+0000","last_undegraded":"2024-03-26T21:22:09.839029+0000","last_fullsized":"2024-03-26T21:22:09.839029+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T04:30:49.939984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494091+0000","last_change":"2024-03-26T21:21:31.210232+0000","last_active":"2024-03-26T21:22:09.494091+0000","last_peered":"2024-03-26T21:22:09.494091+0000","last_clean":"2024-03-26T21:22:09.494091+0000","last_became_active":"2024-03-26T21:21:31.209869+0000","last_became_peered":"2024-03-26T21:21:31.209869+0000","last_unstale":"2024-03-26T21:22:09.494091+0000","last_undegraded":"2024-03-26T21:22:09.494091+0000","last_fullsized":"2024-03-26T21:22:09.494091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T02:35:50.979234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838394+0000","last_change":"2024-03-26T21:21:29.199110+0000","last_active":"2024-03-26T21:22:09.838394+0000","last_peered":"2024-03-26T21:22:09.838394+0000","last_clean":"2024-03-26T21:22:09.838394+0000","last_became_active":"2024-03-26T21:21:29.198790+0000","last_became_peered":"2024-03-26T21:21:29.198790+0000","last_unstale":"2024-03-26T21:22:09.838394+0000","last_undegraded":"2024-03-26T21:22:09.838394+0000","last_fullsized":"2024-03-26T21:22:09.838394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:01:02.970776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.378425+0000","last_change":"2024-03-26T21:21:25.193996+0000","last_active":"2024-03-26T21:22:09.378425+0000","last_peered":"2024-03-26T21:22:09.378425+0000","last_clean":"2024-03-26T21:22:09.378425+0000","last_became_active":"2024-03-26T21:21:25.192736+0000","last_became_peered":"2024-03-26T21:21:25.192736+0000","last_unstale":"2024-03-26T21:22:09.378425+0000","last_undegraded":"2024-03-26T21:22:09.378425+0000","last_fullsized":"2024-03-26T21:22:09.378425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T23:06:45.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.494955+0000","last_change":"2024-03-26T21:21:27.195683+0000","last_active":"2024-03-26T21:22:09.494955+0000","last_peered":"2024-03-26T21:22:09.494955+0000","last_clean":"2024-03-26T21:22:09.494955+0000","last_became_active":"2024-03-26T21:21:27.195230+0000","last_became_peered":"2024-03-26T21:21:27.195230+0000","last_unstale":"2024-03-26T21:22:09.494955+0000","last_undegraded":"2024-03-26T21:22:09.494955+0000","last_fullsized":"2024-03-26T21:22:09.494955+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:35:37.322721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838858+0000","last_change":"2024-03-26T21:21:31.347442+0000","last_active":"2024-03-26T21:22:09.838858+0000","last_peered":"2024-03-26T21:22:09.838858+0000","last_clean":"2024-03-26T21:22:09.838858+0000","last_became_active":"2024-03-26T21:21:31.347054+0000","last_became_peered":"2024-03-26T21:21:31.347054+0000","last_unstale":"2024-03-26T21:22:09.838858+0000","last_undegraded":"2024-03-26T21:22:09.838858+0000","last_fullsized":"2024-03-26T21:22:09.838858+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:30.186355+0000","last_clean_scrub_stamp":"2024-03-26T21:21:30.186355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:02:10.676163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838823+0000","last_change":"2024-03-26T21:21:25.188916+0000","last_active":"2024-03-26T21:22:09.838823+0000","last_peered":"2024-03-26T21:22:09.838823+0000","last_clean":"2024-03-26T21:22:09.838823+0000","last_became_active":"2024-03-26T21:21:25.188139+0000","last_became_peered":"2024-03-26T21:21:25.188139+0000","last_unstale":"2024-03-26T21:22:09.838823+0000","last_undegraded":"2024-03-26T21:22:09.838823+0000","last_fullsized":"2024-03-26T21:22:09.838823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:24.174216+0000","last_clean_scrub_stamp":"2024-03-26T21:21:24.174216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T08:49:38.834721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:09.838510+0000","last_change":"2024-03-26T21:21:27.194089+0000","last_active":"2024-03-26T21:22:09.838510+0000","last_peered":"2024-03-26T21:22:09.838510+0000","last_clean":"2024-03-26T21:22:09.838510+0000","last_became_active":"2024-03-26T21:21:27.193627+0000","last_became_peered":"2024-03-26T21:21:27.193627+0000","last_unstale":"2024-03-26T21:22:09.838510+0000","last_undegraded":"2024-03-26T21:22:09.838510+0000","last_fullsized":"2024-03-26T21:22:09.838510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:26.177342+0000","last_clean_scrub_stamp":"2024-03-26T21:21:26.177342+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T06:40:04.185496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T21:22:45.249892+0000","last_change":"2024-03-26T21:21:29.201941+0000","last_active":"2024-03-26T21:22:45.249892+0000","last_peered":"2024-03-26T21:22:45.249892+0000","last_clean":"2024-03-26T21:22:45.249892+0000","last_became_active":"2024-03-26T21:21:29.201651+0000","last_became_peered":"2024-03-26T21:21:29.201651+0000","last_unstale":"2024-03-26T21:22:45.249892+0000","last_undegraded":"2024-03-26T21:22:45.249892+0000","last_fullsized":"2024-03-26T21:22:45.249892+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:21:28.182231+0000","last_clean_scrub_stamp":"2024-03-26T21:21:28.182231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T00:09:54.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1298432,"data_stored":867823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1294336,"data_stored":868116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":741376,"data_stored":278780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1298432,"data_stored":867129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":692224,"data_stored":276983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":277702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018918,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":276928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31864,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713288,"statfs":{"total":95995035648,"available":95962406912,"internally_reserved":0,"allocated":700416,"data_stored":278139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:22:53.790 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T21:22:53.790 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T21:22:53.790 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T21:22:53.790 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T21:22:54.059 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:53 smithi088 podman[76771]: 2024-03-26 21:22:53.965745792 +0000 UTC m=+0.107923551 volume create c1d22cc0076d04c220afc64c272ff995876bd276a485e9bea16543ee5d47ac8e 2024-03-26T21:22:54.059 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:53 smithi088 podman[76771]: 2024-03-26 21:22:53.875634432 +0000 UTC m=+0.017812206 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T21:22:54.059 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 podman[76771]: 2024-03-26 21:22:54.066290846 +0000 UTC m=+0.208468639 container create 6ef3829d10a46fb327b6c7c214ceb85f88ea8f182745d926ff12d6d4fd25d974 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:22:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:22:54.669 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:54] ENGINE Bus STOPPING 2024-03-26T21:22:54.669 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 podman[76771]: 2024-03-26 21:22:54.385429542 +0000 UTC m=+0.527607303 container init 6ef3829d10a46fb327b6c7c214ceb85f88ea8f182745d926ff12d6d4fd25d974 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:22:54.669 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 podman[76771]: 2024-03-26 21:22:54.390777603 +0000 UTC m=+0.532955363 container start 6ef3829d10a46fb327b6c7c214ceb85f88ea8f182745d926ff12d6d4fd25d974 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:22:54.669 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.403Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T21:22:54.669 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.403Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.404Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.88 port=9094 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.404Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.427Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.428Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.430Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:54.430Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 bash[76771]: 6ef3829d10a46fb327b6c7c214ceb85f88ea8f182745d926ff12d6d4fd25d974 2024-03-26T21:22:54.670 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:54 smithi088 systemd[1]: Started Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:22:54.981 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:22:54.981 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:54] ENGINE Bus STOPPED 2024-03-26T21:22:54.981 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:54] ENGINE Bus STARTING 2024-03-26T21:22:54.981 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:54] ENGINE Serving on http://:::9283 2024-03-26T21:22:54.981 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:22:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:22:54] ENGINE Bus STARTED 2024-03-26T21:22:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:22:55.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:22:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:22:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:22:56.019 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:56 smithi088 ceph-mon[39202]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T21:22:56.019 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:56 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:22:56.019 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:56 smithi088 ceph-mon[39202]: Deploying daemon grafana.a on smithi138 2024-03-26T21:22:56.019 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:56 smithi088 ceph-mon[30075]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T21:22:56.019 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:56 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:22:56.019 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:56 smithi088 ceph-mon[30075]: Deploying daemon grafana.a on smithi138 2024-03-26T21:22:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.313+0000 7f4fc9408700 1 -- 172.21.15.88:0/2323170346 >> v1:172.21.15.88:6790/0 conn(0x7f4fc410dd60 legacy=0x7f4fc4110200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.313+0000 7f4fc9408700 1 -- 172.21.15.88:0/2323170346 shutdown_connections 2024-03-26T21:22:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.313+0000 7f4fc9408700 1 -- 172.21.15.88:0/2323170346 >> 172.21.15.88:0/2323170346 conn(0x7f4fc40fcde0 msgr2=0x7f4fc40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.313+0000 7f4fc9408700 1 -- 172.21.15.88:0/2323170346 shutdown_connections 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.313+0000 7f4fc9408700 1 -- 172.21.15.88:0/2323170346 wait complete. 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fc9408700 1 Processor -- start 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fc9408700 1 -- start start 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fc9408700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4fc41a96a0 con 0x7f4fc410dd60 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fc9408700 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4fc41aa8c0 con 0x7f4fc410a1c0 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fc9408700 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4fc41abb00 con 0x7f4fc4101530 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fc2ffd700 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4fc4101530 0x7f4fc4106730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50752/0 (socket says 172.21.15.88:50752) 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fc2ffd700 1 -- 172.21.15.88:0/1152040921 learned_addr learned my addr 172.21.15.88:0/1152040921 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2264093311 0 0) 0x7f4fc41a96a0 con 0x7f4fc410dd60 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fac0035b0 con 0x7f4fc410dd60 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1116821003 0 0) 0x7f4fc41abb00 con 0x7f4fc4101530 2024-03-26T21:22:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fc41a96a0 con 0x7f4fc4101530 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2653948991 0 0) 0x7f4fc41aa8c0 con 0x7f4fc410a1c0 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.314+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fc41abb00 con 0x7f4fc410a1c0 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4184622630 0 0) 0x7f4fac0035b0 con 0x7f4fc410dd60 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4fc41aa8c0 con 0x7f4fc410dd60 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4fbc0016f0 con 0x7f4fc410dd60 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 803826381 0 0) 0x7f4fc41a96a0 con 0x7f4fc4101530 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4fac0035b0 con 0x7f4fc4101530 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4fb4001870 con 0x7f4fc4101530 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2517945837 0 0) 0x7f4fc41abb00 con 0x7f4fc410a1c0 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4fc41a96a0 con 0x7f4fc410a1c0 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4fb8001870 con 0x7f4fc410a1c0 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3455384848 0 0) 0x7f4fc41aa8c0 con 0x7f4fc410dd60 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 >> v1:172.21.15.88:6790/0 conn(0x7f4fc4101530 legacy=0x7f4fc4106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 >> v1:172.21.15.138:6789/0 conn(0x7f4fc410a1c0 legacy=0x7f4fc41a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc41acd40 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc41abd30 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4fc41ac1f0 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f4fbc001950 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.315+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4fbc004ff0 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.316+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb0005120 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.316+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f4fbc01baf0 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.317+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 6288+0+0 (unknown 3414781811 0 0) 0x7f4fbc089040 con 0x7f4fc410dd60 2024-03-26T21:22:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.321+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f4fbc0563e0 con 0x7f4fc410dd60 2024-03-26T21:22:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:56 smithi138 ceph-mon[33706]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T21:22:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:56 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:22:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:56 smithi138 ceph-mon[33706]: Deploying daemon grafana.a on smithi138 2024-03-26T21:22:56.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.564+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4fb0005410 con 0x7f4fc410dd60 2024-03-26T21:22:56.558 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:22:56 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:22:56.405Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00039326s 2024-03-26T21:22:56.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.564+0000 7f4fabfff700 1 -- 172.21.15.88:0/1152040921 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f4fbc05a090 con 0x7f4fc410dd60 2024-03-26T21:22:56.558 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-26T21:22:56.558 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T21:22:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.566+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 >> v1:172.21.15.88:6800/3939589528 conn(0x7f4fac06f730 legacy=0x7f4fac071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.566+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 >> v1:172.21.15.88:6789/0 conn(0x7f4fc410dd60 legacy=0x7f4fc41a7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:22:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.567+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 shutdown_connections 2024-03-26T21:22:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.567+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 >> 172.21.15.88:0/1152040921 conn(0x7f4fc40fcde0 msgr2=0x7f4fc4111260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:22:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.567+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 shutdown_connections 2024-03-26T21:22:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-26T21:22:56.567+0000 7f4fc9408700 1 -- 172.21.15.88:0/1152040921 wait complete. 2024-03-26T21:22:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:57 smithi088 ceph-mon[30075]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1152040921' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T21:22:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:57 smithi088 ceph-mon[39202]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1152040921' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T21:22:57.347 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T21:22:57.347 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T21:22:57.347 INFO:teuthology.run_tasks:Running task workunit... 2024-03-26T21:22:57.361 INFO:tasks.workunit:Pulling workunits from ref 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:22:57.361 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-26T21:22:57.362 DEBUG:teuthology.orchestra.run.smithi088:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-26T21:22:57.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:57 smithi138 ceph-mon[33706]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:22:57.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1152040921' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T21:22:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:22:57.378 INFO:teuthology.orchestra.run.smithi088.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-26T21:22:57.379 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T21:22:57.435 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-26T21:22:57.435 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-26T21:22:57.490 INFO:tasks.workunit:timeout=3h 2024-03-26T21:22:57.490 INFO:tasks.workunit:cleanup=True 2024-03-26T21:22:57.490 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:22:57.545 INFO:tasks.workunit.client.0.smithi088.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-26T21:22:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:22:59 smithi138 ceph-mon[33706]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:22:59 smithi088 ceph-mon[30075]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:22:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:22:59 smithi088 ceph-mon[39202]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:00.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:23:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:23:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:00 smithi138 ceph-mon[33706]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:00 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:00.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:00 smithi088 ceph-mon[39202]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:00.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:00 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:00 smithi088 ceph-mon[30075]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:00 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:01.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:02 smithi138 ceph-mon[33706]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:02.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:02 smithi088 ceph-mon[39202]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:02.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:02 smithi088 ceph-mon[30075]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:02.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:23:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:23:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:04 smithi138 ceph-mon[33706]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:04.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:04 smithi088 ceph-mon[39202]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:04.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:04 smithi088 ceph-mon[30075]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:04.734 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:04 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:23:04.408Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003041093s 2024-03-26T21:23:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:06 smithi138 ceph-mon[33706]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:06.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:06 smithi088 ceph-mon[39202]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:06 smithi088 ceph-mon[30075]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:08 smithi138 ceph-mon[33706]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:08.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:08 smithi088 ceph-mon[30075]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:08.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:08 smithi088 ceph-mon[39202]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:10.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:23:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:23:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:10 smithi138 ceph-mon[33706]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:10 smithi088 ceph-mon[39202]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:10.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:10 smithi088 ceph-mon[30075]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:10.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:11.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:11.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:12.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:12 smithi088 ceph-mon[39202]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:12 smithi088 ceph-mon[30075]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:12.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:23:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:23:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:12 smithi138 ceph-mon[33706]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:14.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:14 smithi138 ceph-mon[33706]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:14.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:14 smithi088 ceph-mon[30075]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:14.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:14 smithi088 ceph-mon[39202]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:16.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:16 smithi138 ceph-mon[33706]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:16.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:16 smithi088 ceph-mon[30075]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:16.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:16 smithi088 ceph-mon[39202]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:17.185 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:16 smithi138 systemd[1]: Starting Ceph grafana.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:23:17.455 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 podman[66594]: 2024-03-26 21:23:17.096280424 +0000 UTC m=+0.018653751 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-26T21:23:17.455 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 podman[66594]: 2024-03-26 21:23:17.218583838 +0000 UTC m=+0.140957162 container create db10ec6efeae2ed2608a3a83e5300e26e4264a7a24f4aaa2ba9d9b6d27f38fad (image=quay.io/ceph/grafana:9.4.12, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T21:23:17.455 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 podman[66594]: 2024-03-26 21:23:17.462870751 +0000 UTC m=+0.385244080 container init db10ec6efeae2ed2608a3a83e5300e26e4264a7a24f4aaa2ba9d9b6d27f38fad (image=quay.io/ceph/grafana:9.4.12, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T21:23:17.706 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 podman[66594]: 2024-03-26 21:23:17.468421043 +0000 UTC m=+0.390794368 container start db10ec6efeae2ed2608a3a83e5300e26e4264a7a24f4aaa2ba9d9b6d27f38fad (image=quay.io/ceph/grafana:9.4.12, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 bash[66594]: db10ec6efeae2ed2608a3a83e5300e26e4264a7a24f4aaa2ba9d9b6d27f38fad 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 systemd[1]: Started Ceph grafana.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619039062Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619292402Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619309862Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619315748Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619320641Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619325239Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619332896Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619337479Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619341971Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619346876Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619351162Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T21:23:17.707 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619355408Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T21:23:17.708 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619359797Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T21:23:17.708 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619364169Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T21:23:17.708 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619368438Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T21:23:17.708 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619372655Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T21:23:17.708 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619376475Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619381462Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=settings t=2024-03-26T21:23:17.619385851Z level=info msg="App mode production" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=sqlstore t=2024-03-26T21:23:17.619442116Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=sqlstore t=2024-03-26T21:23:17.61945846Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.620687388Z level=info msg="Starting DB migrations" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.620804603Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.621614353Z level=info msg="Executing migration" id="create user table" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.622165164Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.622705034Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.623533196Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.62436925Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.625114154Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.626085263Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.626869259Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T21:23:17.709 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.627698729Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.628634538Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.629246561Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.63014665Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.630758546Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.631526947Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.631976112Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.632439303Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.633302109Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.634046191Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.634918356Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.636387473Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.637168436Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.637979021Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.638887072Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T21:23:17.710 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.639605239Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.640596301Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.641291125Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.642250881Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.642922329Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.643724302Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.644568294Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.645399893Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.64620533Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.647043953Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.649552662Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.652249309Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.65509132Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.657470574Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.660258991Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.660655276Z level=info msg="Executing migration" id="create star table" 2024-03-26T21:23:17.711 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.663339556Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.663792025Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.666030652Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.668624107Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.671374162Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.673974557Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.676414013Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.67682293Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.67952078Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.682148645Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.6846878Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.685125709Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.687739568Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.690547746Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.693000373Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.695672088Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T21:23:17.712 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.698333085Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T21:23:17.713 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.700940729Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T21:23:17.713 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.70355769Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T21:23:17.713 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.706190626Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T21:23:17.713 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.709101396Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T21:23:17.713 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.711425682Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T21:23:17.713 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.714039977Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T21:23:17.959 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.716627489Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T21:23:17.959 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.719063151Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T21:23:17.959 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.72164351Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T21:23:17.959 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.72412187Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.726801864Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.729441032Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.730352634Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.731034062Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.731965188Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.732613709Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.733445861Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.734277167Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.735083481Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.735903119Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.736692978Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.737616163Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.738325474Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.739082369Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T21:23:17.960 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.740031961Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.740656214Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.741738672Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.742572855Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.743347943Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.744144811Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.7449145Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.745736993Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.746706715Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.747348054Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.748146402Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.748978576Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T21:23:17.961 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.749795363Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T21:23:17.962 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.750683941Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T21:23:17.962 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.751397613Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T21:23:17.962 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.7521744Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T21:23:17.962 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.753021732Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T21:23:17.962 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.753879169Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T21:23:17.962 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.754588805Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T21:23:17.962 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.755767851Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T21:23:17.963 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.756712985Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.757498456Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.75818648Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.761102786Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.763775935Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.766590178Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.769166082Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.77176475Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.774379943Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.777069794Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.777386198Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.777928179Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.778285999Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.778719055Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T21:23:17.964 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.77916582Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.779586081Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.780343767Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.781137039Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.782095356Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.782631642Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.783001263Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.783400626Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.784706552Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.785125449Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.785563935Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.786022941Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.786435317Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.786747267Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.78712931Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T21:23:17.965 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.787413035Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.787984972Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.788575927Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.788851978Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.789454021Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.790026252Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.790417793Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.790763947Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.791199767Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.791611167Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.792016807Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.792426379Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.792703877Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.792966594Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T21:23:17.966 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.793585581Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.794178604Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.794472834Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.794860791Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.795302608Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.795603787Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.795995934Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.796413414Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.797006909Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.797318502Z level=info msg="Executing migration" id="create session table" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.797734643Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.798015258Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.798316495Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.798741862Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.799131719Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T21:23:17.967 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.799408642Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.799680506Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.799952359Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.800275104Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.800695716Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.800966211Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.801611432Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.801905693Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.802549053Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.80323421Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.80353324Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.803961886Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.804440379Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.80536822Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.806044515Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.806989684Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T21:23:17.968 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.807662224Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.808562595Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.809230359Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.810052065Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.811721845Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.812503854Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.812959546Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.81374276Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.81473117Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.815412446Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.816132902Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.816939677Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.81954757Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.82237531Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.822841247Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.825792319Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T21:23:17.969 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.828747259Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.83127193Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.831759425Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.834628315Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.837103216Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.839661158Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.840447069Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.843017917Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.845668371Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.848604538Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.849097742Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.851574863Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.854261733Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.856917541Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.857372674Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T21:23:17.970 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.859850523Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.860325545Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.862410361Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.865022489Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.86545706Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.86807444Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.870643476Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.873173583Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.873984215Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.87657001Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.879037135Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.88174205Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.88220276Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.884713049Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.887273649Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T21:23:17.971 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.889710682Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.890239157Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.891122757Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.891823138Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.892689556Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.893628009Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.894434203Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.895242464Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.89597592Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.896316711Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.897132458Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.898081868Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.89872945Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.899117542Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.899900645Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.900684323Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T21:23:17.972 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.901397099Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.901838979Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.902659826Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.903496039Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.904436517Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.90495993Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.905442178Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.906206423Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.907082003Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.907939806Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.908639254Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.909405561Z level=info msg="Executing migration" id="create team table" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.909809486Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.910275151Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.910930316Z level=info msg="Executing migration" id="create team member table" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.911760954Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T21:23:17.973 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.912556422Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.913543105Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.914058862Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.916849423Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.919419796Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.922060737Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.922486753Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.925269411Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.927791956Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.930271562Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.930797824Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.933334537Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.935792148Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.938186258Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.941023774Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T21:23:17.974 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.943682318Z level=info msg="Executing migration" id="create tag table" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.946188146Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.949024661Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.951523362Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.954000988Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.956483303Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.958941129Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.961599602Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.963928087Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.966532115Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.968823895Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.969608629Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.970434639Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T21:23:17.975 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.971251219Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T21:23:18.209 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.972091786Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.972931534Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.973789367Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.974664946Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.975443667Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.976332994Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.977123373Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.977917056Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.978727755Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.979512315Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.980466805Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.981160894Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.982037459Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T21:23:18.210 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.982827084Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T21:23:18.211 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.983770368Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T21:23:18.211 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.984440763Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T21:23:18.211 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.985226549Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T21:23:18.211 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.986006709Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T21:23:18.211 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.986979771Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.987638799Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.988546651Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.989261891Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.990060806Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.990907963Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.991709363Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.992500855Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.993313077Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.994268929Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T21:23:18.212 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.995089154Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.995916942Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.996842366Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:17.997514734Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.000394681Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.000966777Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.003135555Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.00569923Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.008402348Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.011072982Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.013713528Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.016480934Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.019047542Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.023509286Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.02796888Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T21:23:18.213 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.030584327Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.033362344Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.036071642Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.038503681Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.04118416Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.043921994Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.046505241Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.049037882Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.05169075Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.054433059Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.057072353Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.059641137Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.062177032Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.064782028Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T21:23:18.214 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.067327774Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.069952495Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.072586324Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.075043202Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.077748659Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.08028077Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.083004592Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.085600086Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.08819429Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.090556639Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.091589112Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.092564371Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.093536154Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.094552876Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.095453003Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.096162778Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T21:23:18.215 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.097133804Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.097935876Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.098973522Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.099733188Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.100739587Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.10152596Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.102314114Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.103326973Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.104108563Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.104911315Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.105816262Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.10649031Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.107292365Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.10821441Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.108918069Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T21:23:18.216 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.109005943Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.109728969Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.110625485Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.111351642Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.111510088Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.11163507Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.112153395Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.112963561Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.11379787Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.1146159Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.115526291Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.116221887Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.117054606Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.11779786Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.118638951Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.119463687Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T21:23:18.217 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.124956503Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.127459936Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.13033148Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.136100797Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.141749541Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.144320683Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.144841976Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.145223234Z level=info msg="Executing migration" id="create permission table" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.145658257Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.146148589Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.146633781Z level=info msg="Executing migration" id="create role table" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.147068733Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.148138308Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.149203277Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.149668062Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T21:23:18.218 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.150155948Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.150606724Z level=info msg="Executing migration" id="create team role table" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.151032726Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.151494244Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.151988463Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.152448396Z level=info msg="Executing migration" id="create user role table" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.152858451Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.153326881Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.153788627Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T21:23:18.219 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.154259994Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.154705875Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.157300964Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.157776619Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.158901097Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.159408768Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.162004024Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.162485937Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.162905067Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.163325885Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T21:23:18.220 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.163771168Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.164902946Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.165338216Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.165801045Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.166122142Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.166542439Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.166987461Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.167377947Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.167669542Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.168007764Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.168411864Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.168860187Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.170010353Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.170332241Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.170746222Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.173151107Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T21:23:18.221 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.173611547Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.17480461Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.175274954Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.175704971Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.175812105Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.176152064Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.176252557Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.176552Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.179037027Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.179513768Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.181708263Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.18420433Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.18665511Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.189170549Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T21:23:18.222 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.191959968Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.194558419Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.196937419Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.199507962Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.20219545Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.205905818Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.207064618Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.208271006Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.209038939Z level=info msg="Executing migration" id="add share column" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.210238936Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.211037486Z level=info msg="Executing migration" id="create file table" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.21191775Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.212545015Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.213289264Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.213723515Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T21:23:18.223 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.21469147Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.215289632Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.216044412Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.216937721Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.217621543Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.218800997Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.21958311Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.220384761Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T21:23:18.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.22119444Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.222011305Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.222837097Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.224192127Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.225998043Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.226839119Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.243000915Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.243742479Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T21:23:18.475 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.244569723Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.248802566Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.249636301Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.250417305Z level=info msg="Executing migration" id="create folder table" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.2511922Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.252159252Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.252878525Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.253800029Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=migrator t=2024-03-26T21:23:18.25452707Z level=info msg="migrations completed" performed=485 skipped=0 duration=633.763145ms 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=sqlstore t=2024-03-26T21:23:18.254924815Z level=info msg="Created default organization" 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=plugin.finder t=2024-03-26T21:23:18.286724838Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=secrets t=2024-03-26T21:23:18.286809419Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T21:23:18.476 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=query_data t=2024-03-26T21:23:18.290279219Z level=info msg="Query Service initialization" 2024-03-26T21:23:18.477 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=live.push_http t=2024-03-26T21:23:18.29273885Z level=info msg="Live Push Gateway initialization" 2024-03-26T21:23:18.884 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-mon[33706]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:18.884 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.884 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.884 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.885 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.885 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-mon[33706]: Deploying daemon node-exporter.a on smithi088 2024-03-26T21:23:18.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[39202]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[39202]: Deploying daemon node-exporter.a on smithi088 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[30075]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:18.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:18 smithi088 ceph-mon[30075]: Deploying daemon node-exporter.a on smithi088 2024-03-26T21:23:18.985 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:18 smithi088 bash[77334]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T21:23:19.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=infra.usagestats.collector t=2024-03-26T21:23:18.892105773Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T21:23:19.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=provisioning.datasources t=2024-03-26T21:23:18.893286256Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T21:23:19.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=provisioning.datasources t=2024-03-26T21:23:18.904206702Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T21:23:19.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=provisioning.alerting t=2024-03-26T21:23:18.91460967Z level=info msg="starting to provision alerting" 2024-03-26T21:23:19.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=provisioning.alerting t=2024-03-26T21:23:18.914619481Z level=info msg="finished to provision alerting" 2024-03-26T21:23:19.225 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=grafanaStorageLogger t=2024-03-26T21:23:18.914761577Z level=info msg="storage starting" 2024-03-26T21:23:19.225 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=ngalert.state.manager t=2024-03-26T21:23:18.91480916Z level=info msg="Warming state cache for startup" 2024-03-26T21:23:19.225 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=ngalert.state.manager t=2024-03-26T21:23:18.918531993Z level=info msg="State cache has been initialized" states=0 duration=3.72087ms 2024-03-26T21:23:19.225 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=ticker t=2024-03-26T21:23:18.918608693Z level=info msg=starting first_tick=2024-03-26T21:23:20Z 2024-03-26T21:23:19.225 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=ngalert.multiorg.alertmanager t=2024-03-26T21:23:18.918626538Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T21:23:19.225 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:23:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=http.server t=2024-03-26T21:23:18.92158097Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T21:23:20.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:20 smithi138 ceph-mon[33706]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:20.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:20 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:20.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:23:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:23:20.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:20 smithi088 ceph-mon[30075]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:20.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:20 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:20.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:20 smithi088 ceph-mon[39202]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:20.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:20 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:21.233 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:20 smithi088 bash[77334]: Getting image source signatures 2024-03-26T21:23:21.233 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:20 smithi088 bash[77334]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T21:23:21.233 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:20 smithi088 bash[77334]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T21:23:21.234 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:20 smithi088 bash[77334]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T21:23:21.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:21.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:21.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:22.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:22 smithi138 ceph-mon[33706]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:22.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:22 smithi088 ceph-mon[39202]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:22 smithi088 ceph-mon[30075]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:22.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:23:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:23:24.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:24 smithi138 ceph-mon[33706]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:24 smithi088 ceph-mon[39202]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:24.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:24 smithi088 ceph-mon[30075]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:25.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:25.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:25.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:25 smithi088 bash[77334]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T21:23:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:26.022 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:25 smithi088 bash[77334]: Writing manifest to image destination 2024-03-26T21:23:26.387 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 podman[77334]: 2024-03-26 21:23:26.029896754 +0000 UTC m=+7.244303732 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T21:23:26.387 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 podman[77334]: 2024-03-26 21:23:26.153173905 +0000 UTC m=+7.367580925 container create 594e469269315244e47f081dc7f3f2647c500becc6096f14db197de3a2a4ecbc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:26.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-mon[39202]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:26.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-mon[30075]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 podman[77334]: 2024-03-26 21:23:26.395059161 +0000 UTC m=+7.609466140 container init 594e469269315244e47f081dc7f3f2647c500becc6096f14db197de3a2a4ecbc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 podman[77334]: 2024-03-26 21:23:26.401020927 +0000 UTC m=+7.615427907 container start 594e469269315244e47f081dc7f3f2647c500becc6096f14db197de3a2a4ecbc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.404Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.404Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.405Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.405Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.405Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T21:23:26.734 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.405Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T21:23:26.735 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T21:23:26.736 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T21:23:26.737 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T21:23:26.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T21:23:26.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a[77385]: ts=2024-03-26T21:23:26.406Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T21:23:26.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 bash[77334]: 594e469269315244e47f081dc7f3f2647c500becc6096f14db197de3a2a4ecbc 2024-03-26T21:23:26.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:23:26 smithi088 systemd[1]: Started Ceph node-exporter.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:23:26.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:26 smithi138 ceph-mon[33706]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:27.743 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:27 smithi138 systemd[1]: Starting Ceph node-exporter.b for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:23:27.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:27 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:27 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:27 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:27 smithi138 ceph-mon[33706]: Deploying daemon node-exporter.b on smithi138 2024-03-26T21:23:27.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[39202]: Deploying daemon node-exporter.b on smithi138 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:27 smithi088 ceph-mon[30075]: Deploying daemon node-exporter.b on smithi138 2024-03-26T21:23:28.224 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:27 smithi138 bash[66893]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T21:23:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:28 smithi138 ceph-mon[33706]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:28.974 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:28 smithi138 bash[66893]: Getting image source signatures 2024-03-26T21:23:28.974 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:28 smithi138 bash[66893]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T21:23:28.974 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:28 smithi138 bash[66893]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T21:23:28.974 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:28 smithi138 bash[66893]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T21:23:28.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:28 smithi088 ceph-mon[30075]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:28 smithi088 ceph-mon[39202]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:30.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:30 smithi138 ceph-mon[33706]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:30.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:23:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:23:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:30 smithi088 ceph-mon[39202]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:30.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:30 smithi088 ceph-mon[30075]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:31.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:31.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:32.474 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 bash[66893]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T21:23:32.474 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 bash[66893]: Writing manifest to image destination 2024-03-26T21:23:32.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:32 smithi088 ceph-mon[30075]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:32 smithi088 ceph-mon[39202]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:32.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:23:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:23:32.962 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-mon[33706]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:32.962 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 podman[66893]: 2024-03-26 21:23:32.6275596 +0000 UTC m=+4.971543880 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T21:23:32.962 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 podman[66893]: 2024-03-26 21:23:32.725923228 +0000 UTC m=+5.069907488 container create 21807d83fb65938fed267db2d486043cc8e202abc6b668f0c685bae55b3db59a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 podman[66893]: 2024-03-26 21:23:32.96946683 +0000 UTC m=+5.313451090 container init 21807d83fb65938fed267db2d486043cc8e202abc6b668f0c685bae55b3db59a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 podman[66893]: 2024-03-26 21:23:32.975532768 +0000 UTC m=+5.319517028 container start 21807d83fb65938fed267db2d486043cc8e202abc6b668f0c685bae55b3db59a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.979Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.979Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.979Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.979Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T21:23:33.225 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T21:23:33.226 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T21:23:33.227 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.980Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.981Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T21:23:33.228 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b[66955]: ts=2024-03-26T21:23:32.981Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T21:23:33.229 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:33 smithi138 bash[66893]: 21807d83fb65938fed267db2d486043cc8e202abc6b668f0c685bae55b3db59a 2024-03-26T21:23:33.229 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 21:23:33 smithi138 systemd[1]: Started Ceph node-exporter.b for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:23:34.425 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:34 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:34 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:34 smithi138 ceph-mon[33706]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:34.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[30075]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:34 smithi088 ceph-mon[39202]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:35 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:35 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:35 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:36 smithi138 ceph-mon[33706]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:36 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:36 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:36 smithi088 ceph-mon[39202]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:36 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:36 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:36 smithi088 ceph-mon[30075]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:36 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:36 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[39202]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[30075]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:23:39.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:23:39.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:38 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:38 smithi138 ceph-mon[33706]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:38 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:38 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:38 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:23:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:38 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:23:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:38 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:39 smithi138 ceph-mon[33706]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T21:23:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:39 smithi138 ceph-mon[33706]: Reconfiguring daemon alertmanager.a on smithi088 2024-03-26T21:23:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:39 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:39 smithi088 ceph-mon[39202]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T21:23:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:39 smithi088 ceph-mon[39202]: Reconfiguring daemon alertmanager.a on smithi088 2024-03-26T21:23:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:39 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:39 smithi088 ceph-mon[30075]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T21:23:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:39 smithi088 ceph-mon[30075]: Reconfiguring daemon alertmanager.a on smithi088 2024-03-26T21:23:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:39 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:40.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:23:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:23:40.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:40 smithi088 ceph-mon[39202]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:40.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:40 smithi088 ceph-mon[30075]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:40 smithi138 ceph-mon[33706]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:42.151 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:42.151 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:42.733 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:42 smithi088 systemd[1]: Stopping Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:23:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:23:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:23:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:42 smithi138 ceph-mon[33706]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:42 smithi088 ceph-mon[39202]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:42 smithi088 ceph-mon[30075]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:43.234 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[76798]: ts=2024-03-26T21:23:42.998Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:23:43.234 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:43 smithi088 podman[78317]: 2024-03-26 21:23:43.019173415 +0000 UTC m=+0.148350667 container died 6ef3829d10a46fb327b6c7c214ceb85f88ea8f182745d926ff12d6d4fd25d974 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:43.673 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:43 smithi088 podman[78317]: 2024-03-26 21:23:43.446357639 +0000 UTC m=+0.575534892 container cleanup 6ef3829d10a46fb327b6c7c214ceb85f88ea8f182745d926ff12d6d4fd25d974 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:43.674 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:43 smithi088 bash[78317]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T21:23:43.983 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:43 smithi088 podman[78336]: 2024-03-26 21:23:43.681235374 +0000 UTC m=+0.662591100 container remove 6ef3829d10a46fb327b6c7c214ceb85f88ea8f182745d926ff12d6d4fd25d974 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:43.983 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:43 smithi088 podman[78336]: 2024-03-26 21:23:43.781939955 +0000 UTC m=+0.763295680 volume remove c1d22cc0076d04c220afc64c272ff995876bd276a485e9bea16543ee5d47ac8e 2024-03-26T21:23:44.334 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T21:23:44.334 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 systemd[1]: Stopped Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:23:44.334 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 systemd[1]: Starting Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:23:44.334 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 podman[78436]: 2024-03-26 21:23:44.134482535 +0000 UTC m=+0.019240102 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T21:23:44.334 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 podman[78436]: 2024-03-26 21:23:44.24959941 +0000 UTC m=+0.134356967 volume create c160c6e406a3ffb795b342d44dc1f57bbd7d0e434ff0eb01688a27ae27e1e627 2024-03-26T21:23:44.334 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 podman[78436]: 2024-03-26 21:23:44.341803797 +0000 UTC m=+0.226561351 container create b92f5c704a604a7435bc1386201add7f64c62446edc46d81cb208a694f045945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-mon[39202]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-mon[30075]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:44.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 podman[78436]: 2024-03-26 21:23:44.69497447 +0000 UTC m=+0.579732027 container init b92f5c704a604a7435bc1386201add7f64c62446edc46d81cb208a694f045945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:44.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 podman[78436]: 2024-03-26 21:23:44.699939324 +0000 UTC m=+0.584696881 container start b92f5c704a604a7435bc1386201add7f64c62446edc46d81cb208a694f045945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:44.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.711Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T21:23:44.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.711Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T21:23:44.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.712Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.88 port=9094 2024-03-26T21:23:44.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.713Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T21:23:44.985 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.736Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:23:44.985 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.737Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:23:44.985 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.739Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T21:23:44.985 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:44.739Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T21:23:44.985 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 bash[78436]: b92f5c704a604a7435bc1386201add7f64c62446edc46d81cb208a694f045945 2024-03-26T21:23:44.985 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:44 smithi088 systemd[1]: Started Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:23:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:44 smithi138 ceph-mon[33706]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:44 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:44 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:45.958 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:45 smithi138 ceph-mon[33706]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T21:23:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:45 smithi088 ceph-mon[39202]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T21:23:46.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:45 smithi088 ceph-mon[30075]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T21:23:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:46 smithi088 ceph-mon[39202]: Reconfiguring daemon prometheus.a on smithi138 2024-03-26T21:23:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:46 smithi088 ceph-mon[39202]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:46.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:46 smithi088 ceph-mon[30075]: Reconfiguring daemon prometheus.a on smithi138 2024-03-26T21:23:46.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:46 smithi088 ceph-mon[30075]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:46.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:46 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:46.714Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000905604s 2024-03-26T21:23:47.121 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:46 smithi138 ceph-mon[33706]: Reconfiguring daemon prometheus.a on smithi138 2024-03-26T21:23:47.121 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:46 smithi138 ceph-mon[33706]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:48.686 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 systemd[1]: Stopping Ceph prometheus.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:23:48.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-mon[33706]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:48.974 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:23:48.974 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T21:23:48.974 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T21:23:48.974 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T21:23:48.974 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.693Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.694Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.694Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[64524]: ts=2024-03-26T21:23:48.694Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T21:23:48.975 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:48 smithi138 podman[67992]: 2024-03-26 21:23:48.70382765 +0000 UTC m=+0.142754249 container died b166b60e3e9ee4384cf835982f181a515be5d660fb19643b03f54c92541bceab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:48 smithi088 ceph-mon[39202]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:48 smithi088 ceph-mon[30075]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:49.327 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 podman[67992]: 2024-03-26 21:23:49.116092009 +0000 UTC m=+0.555018608 container cleanup b166b60e3e9ee4384cf835982f181a515be5d660fb19643b03f54c92541bceab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:49.327 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 bash[67992]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T21:23:49.724 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 podman[68011]: 2024-03-26 21:23:49.334379701 +0000 UTC m=+0.633307761 container remove b166b60e3e9ee4384cf835982f181a515be5d660fb19643b03f54c92541bceab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:49.724 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T21:23:49.724 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 systemd[1]: Stopped Ceph prometheus.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:23:49.724 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.197s CPU time. 2024-03-26T21:23:49.724 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 systemd[1]: Starting Ceph prometheus.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:23:50.038 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 podman[68109]: 2024-03-26 21:23:49.68864621 +0000 UTC m=+0.018703616 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T21:23:50.039 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:49 smithi138 podman[68109]: 2024-03-26 21:23:49.819051023 +0000 UTC m=+0.149108421 container create 25814fe936c38eb9e2111cb1cc0182c361a8fa4b0aeebc79d09e96cd6378629a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:50.474 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 podman[68109]: 2024-03-26 21:23:50.046381934 +0000 UTC m=+0.376439329 container init 25814fe936c38eb9e2111cb1cc0182c361a8fa4b0aeebc79d09e96cd6378629a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 podman[68109]: 2024-03-26 21:23:50.051818439 +0000 UTC m=+0.381875835 container start 25814fe936c38eb9e2111cb1cc0182c361a8fa4b0aeebc79d09e96cd6378629a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.075Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.075Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.075Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi138 (none))" 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.075Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.075Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.077Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.077Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.078Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.078Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.079Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.079Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.386µs 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.079Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.080Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T21:23:50.475 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.081Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.081Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=16.896µs wal_replay_duration=1.835321ms wbl_replay_duration=416ns total_replay_duration=1.870437ms 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.082Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.082Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.082Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.102Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.105665ms db_storage=713ns remote_storage=1.154µs web_handler=344ns query_engine=497ns scrape=455.633µs scrape_sd=88.401µs notify=10.255µs notify_sd=12.262µs rules=19.264007ms tracing=5.491µs 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.102Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:23:50.102Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 bash[68109]: 25814fe936c38eb9e2111cb1cc0182c361a8fa4b0aeebc79d09e96cd6378629a 2024-03-26T21:23:50.476 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:23:50 smithi138 systemd[1]: Started Ceph prometheus.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STOPPING 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STOPPED 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STARTING 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Serving on http://:::9283 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STARTED 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STOPPING 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STOPPED 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STARTING 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Serving on http://:::9283 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STARTED 2024-03-26T21:23:50.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STOPPING 2024-03-26T21:23:50.485 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:23:50.485 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STOPPED 2024-03-26T21:23:50.485 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STARTING 2024-03-26T21:23:50.921 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Serving on http://:::9283 2024-03-26T21:23:50.922 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: [26/Mar/2024:21:23:50] ENGINE Bus STARTED 2024-03-26T21:23:50.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:23:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:23:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:23:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:23:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:23:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:51.927 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:23:51.927 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:23:51.927 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:23:51.927 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:23:51.928 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:23:51.928 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:23:51.928 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:51 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:23:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:51 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:23:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:51 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:23:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:51 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:23:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:51 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:23:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:51 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:23:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:23:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:23:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:23:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:23:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:23:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:23:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:23:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:52 smithi138 ceph-mon[33706]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:52 smithi088 ceph-mon[30075]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:52 smithi088 ceph-mon[39202]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:54.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[30075]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:54.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:54.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:54.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:54.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[39202]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:54.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:54.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:54.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:54.984 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:23:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:23:54.717Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003970902s 2024-03-26T21:23:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:54 smithi138 ceph-mon[33706]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:54 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:23:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:54 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:54 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:56 smithi138 ceph-mon[33706]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:56 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:56 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:23:57.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:23:57.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:56 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[30075]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:23:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:23:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[39202]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:23:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:23:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:23:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:56 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:23:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:23:58 smithi138 ceph-mon[33706]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:23:58 smithi088 ceph-mon[39202]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:23:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:23:58 smithi088 ceph-mon[30075]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:00.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:24:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:24:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:00 smithi138 ceph-mon[33706]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:00 smithi088 ceph-mon[39202]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:00 smithi088 ceph-mon[30075]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:24:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:24:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:24:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:02 smithi138 ceph-mon[33706]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:02 smithi088 ceph-mon[39202]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:02 smithi088 ceph-mon[30075]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:04 smithi138 ceph-mon[33706]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:04 smithi088 ceph-mon[39202]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:04 smithi088 ceph-mon[30075]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:06 smithi138 ceph-mon[33706]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:06 smithi088 ceph-mon[39202]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:06 smithi088 ceph-mon[30075]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:08 smithi138 ceph-mon[33706]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:08 smithi088 ceph-mon[30075]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:08 smithi088 ceph-mon[39202]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:10.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:24:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:24:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:10 smithi138 ceph-mon[33706]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:10 smithi088 ceph-mon[39202]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:10 smithi088 ceph-mon[30075]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:24:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:24:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:24:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:12 smithi138 ceph-mon[33706]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:12 smithi088 ceph-mon[39202]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:12 smithi088 ceph-mon[30075]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:14 smithi138 ceph-mon[33706]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:14 smithi088 ceph-mon[39202]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:14 smithi088 ceph-mon[30075]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:16 smithi138 ceph-mon[33706]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:16 smithi088 ceph-mon[39202]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:16 smithi088 ceph-mon[30075]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:18 smithi088 ceph-mon[30075]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:18 smithi088 ceph-mon[39202]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:18 smithi138 ceph-mon[33706]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:20.924 INFO:tasks.workunit.client.0.smithi088.stderr:Note: switching to '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'. 2024-03-26T21:24:20.924 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-03-26T21:24:20.924 INFO:tasks.workunit.client.0.smithi088.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T21:24:20.924 INFO:tasks.workunit.client.0.smithi088.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: git switch -c 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:Or undo this operation with: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: git switch - 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-03-26T21:24:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:HEAD is now at 42fc52b86b1 Merge branch 'cv-refactor-osd-objectstore' of https://github.com/guits/ceph into wip-guits-testing-2024-03-26-0925 2024-03-26T21:24:20.930 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-26T21:24:20.962 INFO:tasks.workunit.client.0.smithi088.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T21:24:20.964 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T21:24:20.964 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T21:24:20.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:24:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:24:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:20 smithi088 ceph-mon[39202]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:20 smithi088 ceph-mon[30075]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:20 smithi138 ceph-mon[33706]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:24:21.776 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T21:24:21.846 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T21:24:21.887 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T21:24:21.889 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T21:24:21.889 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T21:24:21.929 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T21:24:21.932 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-26T21:24:21.933 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-26T21:24:21.989 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-26T21:24:21.991 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-26T21:24:21.991 DEBUG:teuthology.orchestra.run.smithi088:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-26T21:24:22.058 INFO:tasks.workunit.client.0.smithi088.stderr:+ parallel=1 2024-03-26T21:24:22.058 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' '' = --serial ']' 2024-03-26T21:24:22.058 INFO:tasks.workunit.client.0.smithi088.stderr:+ crimson=0 2024-03-26T21:24:22.058 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' '' = --crimson ']' 2024-03-26T21:24:22.058 INFO:tasks.workunit.client.0.smithi088.stderr:+ color= 2024-03-26T21:24:22.058 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -t 1 ']' 2024-03-26T21:24:22.059 INFO:tasks.workunit.client.0.smithi088.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-26T21:24:22.059 INFO:tasks.workunit.client.0.smithi088.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T21:24:22.059 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T21:24:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:+ declare -A pids 2024-03-26T21:24:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_aio 2024-03-26T21:24:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_aio' 2024-03-26T21:24:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_aio 2024-03-26T21:24:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.061 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_aio 2024-03-26T21:24:22.061 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79147 2024-03-26T21:24:22.061 INFO:tasks.workunit.client.0.smithi088.stdout:test api_aio on pid 79147 2024-03-26T21:24:22.064 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_aio on pid 79147' 2024-03-26T21:24:22.064 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79147 2024-03-26T21:24:22.064 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.064 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_aio_pp 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_aio_pp' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_aio_pp 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.065 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.066 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.067 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.067 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.067 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.067 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_aio_pp 2024-03-26T21:24:22.067 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79154 2024-03-26T21:24:22.067 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_aio_pp on pid 79154' 2024-03-26T21:24:22.067 INFO:tasks.workunit.client.0.smithi088.stdout:test api_aio_pp on pid 79154 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79154 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_io 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_io' 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_io 2024-03-26T21:24:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.075 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.075 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.075 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_io 2024-03-26T21:24:22.075 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79163 2024-03-26T21:24:22.075 INFO:tasks.workunit.client.0.smithi088.stdout:test api_io on pid 79163 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_io on pid 79163' 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79163 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_io_pp 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_io_pp' 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_io_pp 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_io_pp 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79175 2024-03-26T21:24:22.083 INFO:tasks.workunit.client.0.smithi088.stdout:test api_io_pp on pid 79175 2024-03-26T21:24:22.090 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_io_pp on pid 79175' 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79175 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_asio 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_asio' 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_asio 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.094 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.095 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.098 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.099 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.099 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.099 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.099 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_asio 2024-03-26T21:24:22.099 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79190 2024-03-26T21:24:22.099 INFO:tasks.workunit.client.0.smithi088.stdout:test api_asio on pid 79190 2024-03-26T21:24:22.105 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_asio on pid 79190' 2024-03-26T21:24:22.105 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79190 2024-03-26T21:24:22.105 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.105 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_list 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_list' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_list 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.106 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.107 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.108 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.109 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79147/exe 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.111 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.112 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_list 2024-03-26T21:24:22.113 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.113 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79205 2024-03-26T21:24:22.113 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_list on pid 79205' 2024-03-26T21:24:22.113 INFO:tasks.workunit.client.0.smithi088.stdout:test api_list on pid 79205 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79205 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_lock 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_lock' 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_lock 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_aio: /' 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.121 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.122 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.123 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.123 INFO:tasks.workunit.client.0.smithi088.stdout:test api_lock on pid 79224 2024-03-26T21:24:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_lock 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79154/exe 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79224 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_lock on pid 79224' 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79224 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_lock_pp 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.128 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_lock_pp' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_lock_pp 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.129 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.130 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.131 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.132 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.133 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.134 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.135 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.136 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79163/exe 2024-03-26T21:24:22.137 INFO:tasks.workunit.client.0.smithi088.stdout:test api_lock_pp on pid 79245 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_lock_pp 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79245 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_lock_pp on pid 79245' 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79245 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_misc 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.139 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_misc' 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_misc 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.140 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.141 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.142 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_io.log 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.143 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.144 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_io: /' 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.145 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79175/exe 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_misc 2024-03-26T21:24:22.147 INFO:tasks.workunit.client.0.smithi088.stdout:test api_misc on pid 79265 2024-03-26T21:24:22.150 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79265 2024-03-26T21:24:22.150 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_misc on pid 79265' 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79265 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_misc_pp 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_misc_pp' 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_misc_pp 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.152 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.155 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.155 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_misc_pp 2024-03-26T21:24:22.155 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79278 2024-03-26T21:24:22.155 INFO:tasks.workunit.client.0.smithi088.stdout:test api_misc_pp on pid 79278 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_misc_pp on pid 79278' 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79278 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_tier_pp 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_tier_pp' 2024-03-26T21:24:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_tier_pp 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.163 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.165 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.165 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.165 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.165 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.165 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.165 INFO:tasks.workunit.client.0.smithi088.stdout:test api_tier_pp on pid 79288 2024-03-26T21:24:22.167 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.167 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_tier_pp 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79288 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_tier_pp on pid 79288' 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79288 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.169 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_pool 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_pool' 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.172 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_pool 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79224/exe 2024-03-26T21:24:22.174 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.175 INFO:tasks.workunit.client.0.smithi088.stdout:test api_pool on pid 79305 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_pool 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79305 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_pool on pid 79305' 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79305 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_snapshots 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.177 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_snapshots' 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.178 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.180 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_lock: /' 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_snapshots 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.185 INFO:tasks.workunit.client.0.smithi088.stdout:test api_snapshots on pid 79322 2024-03-26T21:24:22.187 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_snapshots 2024-03-26T21:24:22.187 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79322 2024-03-26T21:24:22.187 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_snapshots on pid 79322' 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79322 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_snapshots_pp 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_snapshots_pp' 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_snapshots_pp 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.189 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.190 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.192 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.194 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.195 INFO:tasks.workunit.client.0.smithi088.stdout:test api_snapshots_pp on pid 79339 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_snapshots_pp 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79339 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_snapshots_pp on pid 79339' 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79339 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.201 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.202 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.204 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79265/exe 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79190/exe 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.206 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.207 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.209 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.210 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79205/exe 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_stat 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_stat' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.212 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_asio: /' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_misc: /' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.213 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-26T21:24:22.214 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.215 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_list.log 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_list: /' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.216 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_stat 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.217 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.222 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_stat 2024-03-26T21:24:22.223 INFO:tasks.workunit.client.0.smithi088.stdout:test api_stat on pid 79441 2024-03-26T21:24:22.227 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79441 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_stat on pid 79441' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79441 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.228 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_stat_pp 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.229 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_stat_pp' 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.230 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.231 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.232 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_stat_pp 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79288/exe 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.233 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.234 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.235 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79245/exe 2024-03-26T21:24:22.238 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79305/exe 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79322/exe 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.239 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.240 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.242 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.242 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.242 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.242 INFO:tasks.workunit.client.0.smithi088.stdout:test api_stat_pp on pid 79504 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_stat_pp 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79504 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_stat_pp on pid 79504' 2024-03-26T21:24:22.244 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79504 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-26T21:24:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_watch_notify 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_watch_notify' 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_watch_notify 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_pool: /' 2024-03-26T21:24:22.248 INFO:tasks.workunit.client.0.smithi088.stdout:test api_watch_notify on pid 79527 2024-03-26T21:24:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_watch_notify 2024-03-26T21:24:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79527 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_watch_notify on pid 79527' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79527 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.255 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.256 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79278/exe 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.257 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.258 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79339/exe 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.259 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.260 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.261 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_watch_notify_pp 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_watch_notify_pp' 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-26T21:24:22.264 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_watch_notify_pp 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.265 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.266 INFO:tasks.workunit.client.0.smithi088.stdout:test api_watch_notify_pp on pid 79562 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_watch_notify_pp 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79562 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_watch_notify_pp on pid 79562' 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79562 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_cmd 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_cmd' 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_cmd 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.270 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.271 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.271 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.273 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.274 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.276 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.277 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79441/exe 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.280 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.280 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-26T21:24:22.280 INFO:tasks.workunit.client.0.smithi088.stdout:test api_cmd on pid 79598 2024-03-26T21:24:22.283 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_cmd 2024-03-26T21:24:22.283 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79598 2024-03-26T21:24:22.283 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_cmd on pid 79598' 2024-03-26T21:24:22.283 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79598 2024-03-26T21:24:22.283 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79504/exe 2024-03-26T21:24:22.286 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_stat: /' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.287 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_cmd_pp 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_cmd_pp' 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.288 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.289 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.290 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_cmd_pp 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.291 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79527/exe 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.295 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.296 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.296 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-26T21:24:22.296 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-26T21:24:22.296 INFO:tasks.workunit.client.0.smithi088.stdout:test api_cmd_pp on pid 79712 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_cmd_pp 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79712 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_cmd_pp on pid 79712' 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79712 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-26T21:24:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_service 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_service' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.300 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_service 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.302 INFO:tasks.workunit.client.0.smithi088.stdout:test api_service on pid 79735 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_service 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79735 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_service on pid 79735' 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79735 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.304 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.304 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.304 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.304 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:22.304 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.306 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.308 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_service_pp 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_service_pp' 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.309 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.310 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_service_pp 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.311 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79562/exe 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.313 INFO:tasks.workunit.client.0.smithi088.stdout:test api_service_pp on pid 79774 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_service_pp 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79774 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_service_pp on pid 79774' 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79774 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.317 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_c_write_operations 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.318 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_c_write_operations' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_c_write_operations 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79598/exe 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.323 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_c_write_operations 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79825 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_c_write_operations on pid 79825' 2024-03-26T21:24:22.324 INFO:tasks.workunit.client.0.smithi088.stdout:test api_c_write_operations on pid 79825 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79825 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_c_read_operations 2024-03-26T21:24:22.327 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_cmd: /' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_c_read_operations' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_c_read_operations 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79712/exe 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.332 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_c_read_operations 2024-03-26T21:24:22.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79850 2024-03-26T21:24:22.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_c_read_operations on pid 79850' 2024-03-26T21:24:22.334 INFO:tasks.workunit.client.0.smithi088.stdout:test api_c_read_operations on pid 79850 2024-03-26T21:24:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79850 2024-03-26T21:24:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-26T21:24:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_cls_remote_reads 2024-03-26T21:24:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_cls_remote_reads' 2024-03-26T21:24:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_cls_remote_reads 2024-03-26T21:24:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.343 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.344 INFO:tasks.workunit.client.0.smithi088.stdout:test api_cls_remote_reads on pid 79875 2024-03-26T21:24:22.346 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_cls_remote_reads 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79875 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_cls_remote_reads on pid 79875' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79875 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s list_parallel 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' list_parallel' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo list_parallel 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.351 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.352 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.352 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.352 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.352 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.352 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.352 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.352 INFO:tasks.workunit.client.0.smithi088.stdout:test list_parallel on pid 79895 2024-03-26T21:24:22.356 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=list_parallel 2024-03-26T21:24:22.356 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79895 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test list_parallel on pid 79895' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79895 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79735/exe 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s open_pools_parallel 2024-03-26T21:24:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' open_pools_parallel' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo open_pools_parallel 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.364 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.365 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.365 INFO:tasks.workunit.client.0.smithi088.stdout:test open_pools_parallel on pid 79921 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=open_pools_parallel 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79921 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test open_pools_parallel on pid 79921' 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79921 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s delete_pools_parallel 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' delete_pools_parallel' 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.371 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo delete_pools_parallel 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.372 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.373 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_service.log 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.379 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=delete_pools_parallel 2024-03-26T21:24:22.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79774/exe 2024-03-26T21:24:22.379 INFO:tasks.workunit.client.0.smithi088.stdout:test delete_pools_parallel on pid 79954 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79954 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test delete_pools_parallel on pid 79954' 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79954 2024-03-26T21:24:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_service: /' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79850/exe 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s cls 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' cls' 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.384 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo cls 2024-03-26T21:24:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.388 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-26T21:24:22.393 INFO:tasks.workunit.client.0.smithi088.stdout:test cls on pid 79989 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=cls 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79989 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test cls on pid 79989' 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=79989 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-26T21:24:22.401 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s cmd 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' cmd' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.402 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.403 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo cmd 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.406 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79895/exe 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79875/exe 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79825/exe 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.413 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.414 INFO:tasks.workunit.client.0.smithi088.stdout:test cmd on pid 80037 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=cmd 2024-03-26T21:24:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80037 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test cmd on pid 80037' 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80037 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s handler_error 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' handler_error' 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.421 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-26T21:24:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo handler_error 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.424 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.424 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.424 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.424 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.424 INFO:tasks.workunit.client.0.smithi088.stdout:test handler_error on pid 80051 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=handler_error 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80051 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test handler_error on pid 80051' 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80051 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s io 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' io' 2024-03-26T21:24:22.428 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo io 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ list_parallel: /' 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.429 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79921/exe 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79954/exe 2024-03-26T21:24:22.437 INFO:tasks.workunit.client.0.smithi088.stdout:test io on pid 80095 2024-03-26T21:24:22.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=io 2024-03-26T21:24:22.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80095 2024-03-26T21:24:22.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test io on pid 80095' 2024-03-26T21:24:22.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80095 2024-03-26T21:24:22.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.440 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s ec_io 2024-03-26T21:24:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' ec_io' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo ec_io 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.444 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.445 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.446 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.447 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.448 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.448 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.448 INFO:tasks.workunit.client.0.smithi088.stdout:test ec_io on pid 80146 2024-03-26T21:24:22.451 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=ec_io 2024-03-26T21:24:22.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80146 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test ec_io on pid 80146' 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80146 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s list 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' list' 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/79989/exe 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.455 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_cls 2024-03-26T21:24:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_cls.log 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ cls: /' 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80051/exe 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.461 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo list 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_handler_error 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.462 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.463 INFO:tasks.workunit.client.0.smithi088.stdout:test list on pid 80239 2024-03-26T21:24:22.467 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.467 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.467 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.467 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.467 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.467 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=list 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.468 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80239 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test list on pid 80239' 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80239 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s ec_list 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' ec_list' 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.469 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ handler_error: /' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.471 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo ec_list 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.472 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.473 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.474 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.475 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.476 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.477 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.478 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80095/exe 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.479 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.480 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.481 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80146/exe 2024-03-26T21:24:22.482 INFO:tasks.workunit.client.0.smithi088.stdout:test ec_list on pid 80343 2024-03-26T21:24:22.484 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:22.485 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.485 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=ec_list 2024-03-26T21:24:22.485 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80343 2024-03-26T21:24:22.485 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test ec_list on pid 80343' 2024-03-26T21:24:22.485 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80343 2024-03-26T21:24:22.485 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.485 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_ec_io 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ ec_io: /' 2024-03-26T21:24:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80037/exe 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s misc 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' misc' 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_cmd 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ cmd: /' 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo misc 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.488 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.490 INFO:tasks.workunit.client.0.smithi088.stdout:test misc on pid 80405 2024-03-26T21:24:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=misc 2024-03-26T21:24:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80405 2024-03-26T21:24:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test misc on pid 80405' 2024-03-26T21:24:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80405 2024-03-26T21:24:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.493 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s pool 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' pool' 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_io 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_io.log 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ io: /' 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo pool 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=pool 2024-03-26T21:24:22.499 INFO:tasks.workunit.client.0.smithi088.stdout:test pool on pid 80439 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80439 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test pool on pid 80439' 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80439 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.503 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.504 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s read_operations 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' read_operations' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.505 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.506 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.507 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.508 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo read_operations 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.509 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80239/exe 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.510 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.511 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.513 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.513 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.513 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.513 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.513 INFO:tasks.workunit.client.0.smithi088.stdout:test read_operations on pid 80477 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=read_operations 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80477 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test read_operations on pid 80477' 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80477 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s snapshots 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.517 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' snapshots' 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_list.log 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ list: /' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_list 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo snapshots 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.520 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=snapshots 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.523 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80525 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.524 INFO:tasks.workunit.client.0.smithi088.stdout:test snapshots on pid 80525 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test snapshots on pid 80525' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80525 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s watch_notify 2024-03-26T21:24:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' watch_notify' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80343/exe 2024-03-26T21:24:22.528 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo watch_notify 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.531 INFO:tasks.workunit.client.0.smithi088.stdout:test watch_notify on pid 80556 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=watch_notify 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80556 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test watch_notify on pid 80556' 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80556 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ ec_list: /' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_ec_list 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80439/exe 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s write_operations 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' write_operations' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo write_operations 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80405/exe 2024-03-26T21:24:22.542 INFO:tasks.workunit.client.0.smithi088.stdout:test write_operations on pid 80583 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=write_operations 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80583 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test write_operations on pid 80583' 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=80583 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ ret=0 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:24:22.551 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79288 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79288 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.552 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_misc 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_misc.log 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ misc: /' 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.553 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.554 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_pool 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ pool: /' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_pool.log 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.555 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.556 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.557 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.559 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80525/exe 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80477/exe 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.564 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_snapshots 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ snapshots: /' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ read_operations: /' 2024-03-26T21:24:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_read_operations 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.568 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T21:24:22.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80583/exe 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_write_operations 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ write_operations: /' 2024-03-26T21:24:22.572 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T21:24:22.573 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/80556/exe 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T21:24:22.574 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-03-26T21:24:22.575 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_neorados_watch_notify 2024-03-26T21:24:22.575 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ watch_notify: /' 2024-03-26T21:24:22.575 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-26T21:24:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:24:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:24:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2378569791' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T21:24:23.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:22 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2378569791' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2378569791' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:23.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:22 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:24.403 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] Global test environment set-up. 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (3 ms) 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (47 ms) 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3 ms) 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-26T21:24:24.404 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3 ms) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (4 ms) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] 12 tests from AsioRados (68 ms total) 2024-03-26T21:24:24.405 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-03-26T21:24:24.406 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] Global test environment tear-down 2024-03-26T21:24:24.406 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2211 ms total) 2024-03-26T21:24:24.406 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-26T21:24:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T21:24:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi088-79779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/260143835' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2587594827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3979706969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/122141728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/508100425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2322983644' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3449447425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/386392667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1952429443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/324100550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1495628132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3227422205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/779827424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi088-79971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3720817424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.34859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/65741186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi088-79254-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3378501083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi088-79496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24742 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.34670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/260509449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3130248226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.34862 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.34658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24706 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:24 smithi138 ceph-mon[33706]: from='client.24670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi088-79779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/260143835' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2587594827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3979706969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/122141728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/508100425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2322983644' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3449447425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/386392667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi088-80346-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi088-80549-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T21:24:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi088-79779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/260143835' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2587594827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3979706969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/122141728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/508100425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2322983644' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3449447425' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/386392667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1952429443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/324100550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1495628132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3227422205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/779827424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi088-79971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3720817424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.34859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/65741186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi088-79254-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3378501083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi088-79496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24742 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.34670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/260509449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3130248226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.34862 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.34658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24706 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[30075]: from='client.24670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1952429443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/324100550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1495628132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3227422205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/779827424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi088-79971-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3720817424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.34859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/65741186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi088-79254-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3378501083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi088-79496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24742 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.34670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/260509449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3130248226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.34862 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.34658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24706 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24718 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.491 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:24 smithi088 ceph-mon[39202]: from='client.24670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] Global test environment set-up. 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjects (428 ms) 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (37 ms) 2024-03-26T21:24:24.716 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo2 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo3 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo4 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo5 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo6,foo7 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo7 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo6 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo4 2024-03-26T21:24:24.717 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo5 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo7 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo6 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo1 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo1 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo2 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo3 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (80 ms) 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 1 0 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 10 0 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 13 0 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 7 0 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 14 0 2024-03-26T21:24:24.718 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 0 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 15 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 11 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 5 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 8 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 6 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 3 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 4 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 12 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 9 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2 0 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (69 ms) 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: x cursor=MIN 2024-03-26T21:24:24.719 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=1 cursor=14:02547ec2:::1:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=10 cursor=14:52ea6a34:::10:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=13 cursor=14:566253c9:::13:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=7 cursor=14:5c6b0b28:::7:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=14 cursor=14:62a1935d:::14:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=0 cursor=14:6cac518f:::0:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=15 cursor=14:863748b0:::15:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=11 cursor=14:89d3ae78:::11:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=5 cursor=14:b29083e3:::5:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=8 cursor=14:bd63b0f1:::8:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=6 cursor=14:c4fdafeb:::6:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=3 cursor=14:cfc208b3:::3:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=4 cursor=14:d83876eb:::4:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=12 cursor=14:de5d7c5f:::12:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=9 cursor=14:e960b815:::9:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=2 cursor=14:f905c69b:::2:head 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-26T21:24:24.720 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=1 cursor=14:02547ec2:::1:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:02547ec2:::1:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:02547ec2:::1:head -> 1 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=10 cursor=14:52ea6a34:::10:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:52ea6a34:::10:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:52ea6a34:::10:head -> 10 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=13 cursor=14:566253c9:::13:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:566253c9:::13:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:566253c9:::13:head -> 13 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=7 cursor=14:5c6b0b28:::7:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:5c6b0b28:::7:head 2024-03-26T21:24:24.721 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:5c6b0b28:::7:head -> 7 2024-03-26T21:24:24.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=14 cursor=14:62a1935d:::14:head 2024-03-26T21:24:24.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:62a1935d:::14:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:62a1935d:::14:head -> 14 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=0 cursor=14:6cac518f:::0:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:6cac518f:::0:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:6cac518f:::0:head -> 0 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=15 cursor=14:863748b0:::15:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:863748b0:::15:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:863748b0:::15:head -> 15 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=11 cursor=14:89d3ae78:::11:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:89d3ae78:::11:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:89d3ae78:::11:head -> 11 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=5 cursor=14:b29083e3:::5:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:b29083e3:::5:head 2024-03-26T21:24:24.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:b29083e3:::5:head -> 5 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=8 cursor=14:bd63b0f1:::8:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:bd63b0f1:::8:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:bd63b0f1:::8:head -> 8 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=6 cursor=14:c4fdafeb:::6:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:c4fdafeb:::6:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:c4fdafeb:::6:head -> 6 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=3 cursor=14:cfc208b3:::3:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:cfc208b3:::3:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:cfc208b3:::3:head -> 3 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=4 cursor=14:d83876eb:::4:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:d83876eb:::4:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:d83876eb:::4:head -> 4 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=12 cursor=14:de5d7c5f:::12:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:de5d7c5f:::12:head 2024-03-26T21:24:24.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:de5d7c5f:::12:head -> 12 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=9 cursor=14:e960b815:::9:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:e960b815:::9:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:e960b815:::9:head -> 9 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=2 cursor=14:f905c69b:::2:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:f905c69b:::2:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:f905c69b:::2:head -> 2 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:f905c69b:::2:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:f905c69b:::2:head expected=14:f905c69b:::2:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:f905c69b:::2:head -> 2 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=2 expected=2 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:e960b815:::9:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:e960b815:::9:head expected=14:e960b815:::9:head 2024-03-26T21:24:24.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:e960b815:::9:head -> 9 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=9 expected=9 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:c4fdafeb:::6:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:c4fdafeb:::6:head expected=14:c4fdafeb:::6:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:c4fdafeb:::6:head -> 6 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=6 expected=6 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:bd63b0f1:::8:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:bd63b0f1:::8:head expected=14:bd63b0f1:::8:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:bd63b0f1:::8:head -> 8 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=8 expected=8 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:b29083e3:::5:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:b29083e3:::5:head expected=14:b29083e3:::5:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:b29083e3:::5:head -> 5 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=5 expected=5 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:89d3ae78:::11:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:89d3ae78:::11:head expected=14:89d3ae78:::11:head 2024-03-26T21:24:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:89d3ae78:::11:head -> 11 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=11 expected=11 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:863748b0:::15:head 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:863748b0:::15:head expected=14:863748b0:::15:head 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:863748b0:::15:head -> 15 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=15 expected=15 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:6cac518f:::0:head 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:6cac518f:::0:head expected=14:6cac518f:::0:head 2024-03-26T21:24:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:6cac518f:::0:head -> 0 2024-03-26T21:24:24.932 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=0 list_parallel: process_1_[80088]: starting. 2024-03-26T21:24:24.932 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[80088]: creating pool ceph_test_rados_list_parallel.smithi088-80029 2024-03-26T21:24:24.932 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[80088]: created object 0... 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[80088]: created object 25... 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[80088]: created object 49... 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[80088]: finishing. 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[80088]: shutting down. 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[80089]: starting. 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[80089]: listing objects. 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[80089]: listed object 0... 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[80089]: listed object 25... 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[80089]: saw 50 objects 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[80089]: shutting down. 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[80891]: starting. 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[80891]: creating pool ceph_test_rados_list_parallel.smithi088-80029 2024-03-26T21:24:24.933 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[80891]: created object 0... 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[80891]: created object 25... 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[80891]: created object 49... 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[80891]: finishing. 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[80891]: shutting down. 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[80892]: starting. 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[80892]: listing objects. 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[80892]: listed object 0... 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[80892]: listed object 25... 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[80892]: saw 43 objects 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[80892]: shutting down. 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[80893]: starting. 2024-03-26T21:24:24.934 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[80893]: removed 25 objects... 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[80893]: removed half of the objects 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[80893]: removed 50 objects... 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[80893]: removed 50 objects 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[80893]: shutting down. 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[80939]: starting. 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[80939]: creating pool ceph_test_rados_list_parallel.smithi088-80029 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[80939]: created object 0... 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[80939]: created object 25... 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[80939]: created object 49... 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[80939]: finishing. 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[80939]: shutting down. 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.935 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[80940]: starting. 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[80940]: listing objects. 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[80940]: listed object 0... 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[80940]: listed object 25... 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[80940]: listed object 50... 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[80940]: saw 56 objects 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[80940]: shutting down. 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[80941]: starting. 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[80941]: added 25 objects... 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[80941]: added half of the objects 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[80941]: added 50 objects... 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[80941]: added 50 objects 2024-03-26T21:24:24.936 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[80941]: shutting down. 2024-03-26T21:24:24.937 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:24.937 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ********************** api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (2681 ms) 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (2681 ms total) 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (2681 ms total) 2024-03-26T21:24:25.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (73 ms) 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (51 ms) 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2634 ms) 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2758 ms total) 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2758 ms total) 2024-03-26T21:24:25.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: Running main() from gmock_main.cc 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [----------] Global test environment set-up. 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2634 ms) 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [----------] 1 test from neocls_handler_error (2634 ms total) 2024-03-26T21:24:25.074 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [----------] Global test environment tear-down 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2634 ms total) 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: handler_error: [ PASSED ] 1 test. 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (331 ms) 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (6 ms) 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (14 ms) 2024-03-26T21:24:25.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (29 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (8 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (9 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (8 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-03-26T21:24:25.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (13 ms) 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (10 ms) 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (474 ms total) 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2741 ms total) 2024-03-26T21:24:25.077 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-26T21:24:25.134 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-26T21:24:25.134 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-26T21:24:25.134 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-26T21:24:25.134 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-26T21:24:25.134 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (78 ms) 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (69 ms) 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2606 ms) 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.007127+0000 mon.a [WRN] Health check failed: 35 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018041+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/779827424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi088-79971-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018134+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3720817424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018176+0000 mon.a [INF] from='client.34859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018267+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/65741186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi088-79254-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018344+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3378501083' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi088-79496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018447+0000 mon.a [INF] from='client.24742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018538+0000 mon.a [INF] from='client.34670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.135 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018628+0000 mon.a [INF] from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.136 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018707+0000 mon.a [INF] from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.136 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018788+0000 mon.a [INF] from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.136 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018871+0000 mon.a [INF] from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.018974+0000 mon.a [INF] from='client.34658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP********* 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[80987]: starting. 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[80987]: creating pool ceph_test_rados_list_parallel.smithi088-80029 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[80987]: created object 0... 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[80987]: created object 25... 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[80987]: created object 49... 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[80987]: finishing. 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[80987]: shutting down. 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.434 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: starting. 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: listing objects. 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: listed object 0... 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: listed object 25... 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: listed object 50... 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: listed object 75... 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: saw 98 objects 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[80988]: shutting down. 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[80991]: starting. 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[80991]: removed 25 objects... 2024-03-26T21:24:25.435 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[80991]: removed half of the objects 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[80991]: removed 50 objects... 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[80991]: removed 50 objects 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[80991]: shutting down. 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[80989]: starting. 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[80989]: added 25 objects... 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[80989]: added half of the objects 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[80989]: added 50 objects... 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[80989]: added 50 objects 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[80989]: shutting down. 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.436 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[80990]: starting. 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[80990]: added 25 objects... 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[80990]: added half of the objects 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[80990]: added 50 objects... 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[80990]: added 50 objects 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[80990]: shutting down. 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[81067]: starting. 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[81067]: creating pool ceph_test_rados_list_parallel.smithi088-80029 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[81067]: created object 0... 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[81067]: created object 25... 2024-03-26T21:24:25.437 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[81067]: created object 49... 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[81067]: finishing. 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[81067]: shutting down. 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: starting. 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: listing objects. 2024-03-26T21:24:25.438 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: listed object 0... 2024-03-26T21:24:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: pgmap v71: 1252 pgs: 1120 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:24:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: Health check failed: 35 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/779827424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi088-79971-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3720817424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.34859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/65741186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi088-79254-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3378501083' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi088-79496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.34670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.34658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24706 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.24670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:25.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: pgmap v71: 1252 pgs: 1120 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:24:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: Health check failed: 35 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/779827424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi088-79971-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3720817424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.34859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/65741186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi088-79254-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3378501083' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi088-79496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.34670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.34658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24706 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: pgmap v71: 1252 pgs: 1120 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: Health check failed: 35 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/779827424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi088-79971-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3720817424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.34859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79819-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/65741186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi088-79254-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3378501083' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi088-79496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi088-79627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.34670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi088-79308-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi088-79701-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi088-79544-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.34862 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.34658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24706 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.24670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.24670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:25.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:25.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:26.056 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: listed object 25... 2024-03-26T21:24:26.056 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: listed object 50... 2024-03-26T21:24:26.056 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: listed object 75... 2024-03-26T21:24:26.056 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: listed object 100... 2024-03-26T21:24:26.056 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: listed object 125... 2024-03-26T21:24:26.056 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: saw 150 objects 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[81068]: shutting down. 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[81069]: starting. 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[81069]: added 25 objects... 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[81069]: added half of the objects 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[81069]: added 50 objects... 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[81069]: added 50 objects 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[81069]: shutting down. 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.057 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.058 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.058 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-03-26T21:24:26.058 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******* SUCCESS ********** 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout:_smithi088-79269-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019060+0000 mon.a [INF] from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi088-79235-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019142+0000 mon.a [INF] from='client.34712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi088-79387-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019221+0000 mon.a [INF] from='client.24706 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019300+0000 mon.a [INF] from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi088-79584-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019382+0000 mon.a [INF] from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi088-79509-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019470+0000 mon.a [INF] from='client.24718 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi088-79477-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019569+0000 mon.a [INF] from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.019659+0000 mon.a [INF] from='client.24670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi088-79214-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:24.600942+0000 mon.a [INF] from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.028812+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi088-79779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.028977+0000 mon.a [INF] from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.029090+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.041356+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.055217+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.138854+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.141293+0000 client.admin [INF] onexx 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.161543+0000 mon.a [INF] from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.189291+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.189722+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.194884+0000 mon.a [INF] from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.195635+0000 mon.a [INF] from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.197992+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.204678+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.204921+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.206912+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.215922+0000 mon.a [INF] from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.216941+0000 mon.a [INF] from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.222704+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.225390+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.228614+0000 mon.a [INF] from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.228988+0000 mon.a [INF] from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.231668+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi088-79779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: onexx 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi088-79779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: onexx 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi088-79779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi088-80346-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi088-80549-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: onexx 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:26.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: pgmap v74: 732 pgs: 184 creating+peering, 548 active+clean; 605 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 79 KiB/s wr, 245 op/s 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1629047615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1949366716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:27.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.25078 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2174745510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: twoxx 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:27.378 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:27.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:27.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:27.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:27.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:27.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:27.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:27.379 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:27 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: pgmap v74: 732 pgs: 184 creating+peering, 548 active+clean; 605 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 79 KiB/s wr, 245 op/s 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1629047615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1949366716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: pgmap v74: 732 pgs: 184 creating+peering, 548 active+clean; 605 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 79 KiB/s wr, 245 op/s 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1629047615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1949366716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.25078 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2174745510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: twoxx 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:27.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.25093 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.25078 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2174745510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: twoxx 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T21:24:27.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:27.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:27.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:27.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:27.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:27 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.232911+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smith delete_pools_parallel: process_1_[80124]: starting. 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[80124]: creating pool ceph_test_rados_delete_pools_parallel.smithi088-80105 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[80124]: created object 0... 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[80124]: created object 25... 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[80124]: created object 49... 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[80124]: finishing. 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[80124]: shutting down. 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_2_[80125]: starting. 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_2_[80125]: deleting pool ceph_test_rados_delete_pools_parallel.smithi088-80105 2024-03-26T21:24:28.055 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_2_[80125]: shutting down. 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[81116]: starting. 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[81116]: creating pool ceph_test_rados_delete_pools_parallel.smithi088-80105 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[81116]: created object 0... 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[81116]: created object 25... 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[81116]: created object 49... 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[81116]: finishing. 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[81116]: shutting down. 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[81118]: starting. 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[81118]: listing objects. 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[81118]: listed object 0... 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[81118]: listed object 25... 2024-03-26T21:24:28.056 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[81118]: saw 50 objects 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[81118]: shutting down. 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_4_[81117]: starting. 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_4_[81117]: deleting pool ceph_test_rados_delete_pools_parallel.smithi088-80105 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_4_[81117]: shutting down. 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-03-26T21:24:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: Running main() from gmock_main.cc 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [----------] Global test environment set-up. 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [ OK ] NeoRadosCls.DNE (2627 ms) 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (2995 ms) 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [----------] 2 tests from NeoRadosCls (5622 ms total) 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [----------] Global test environment tear-down 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [==========] 2 tests from 1 test suite ran. (5622 ms total) 2024-03-26T21:24:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: cls: [ PASSED ] 2 tests. 2024-03-26T21:24:28.060 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[80099]: starting. 2024-03-26T21:24:28.060 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[80099]: creating pool ceph_test_rados_open_pools_parallel.smithi088-80077 2024-03-26T21:24:28.060 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[80099]: created object 0... 2024-03-26T21:24:28.060 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[80099]: created object 25... 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[80099]: created object 49... 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[80099]: finishing. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[80099]: shutting down. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[80100]: starting. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[80100]: rados_pool_create. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[80100]: rados_ioctx_create. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[80100]: shutting down. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[81128]: starting. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[81128]: creating pool ceph_test_rados_open_pools_parallel.smithi088-80077 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[81128]: created object 0... 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[81128]: created object 25... 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[81128]: created object 49... 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[81128]: finishing. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[81128]: shutting down. 2024-03-26T21:24:28.061 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-03-26T21:24:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[81132]: starting. 2024-03-26T21:24:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[81132]: rados_pool_create. 2024-03-26T21:24:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[81132]: rados_ioctx_create. 2024-03-26T21:24:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[81132]: shutting down. 2024-03-26T21:24:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-03-26T21:24:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-03-26T21:24:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-26T21:24:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.25078 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2174745510' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: 19.6 deep-scrub starts 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: 19.0 deep-scrub starts 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: 19.6 deep-scrub ok 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2302504126' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: 19.2 deep-scrub starts 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: 19.2 deep-scrub ok 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:28.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:28.479 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:28 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.25078 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2174745510' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.25078 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2174745510' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: 19.6 deep-scrub starts 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: 19.0 deep-scrub starts 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: 19.6 deep-scrub ok 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2302504126' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: 19.2 deep-scrub starts 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: 19.2 deep-scrub ok 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: 19.6 deep-scrub starts 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: 19.0 deep-scrub starts 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: 19.6 deep-scrub ok 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2302504126' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: 19.2 deep-scrub starts 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: 19.2 deep-scrub ok 2024-03-26T21:24:28.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:28.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:28.491 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.492 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:28.493 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:28 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.077 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: Running main() from gmock_main.cc 2024-03-26T21:24:29.077 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-26T21:24:29.077 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [----------] Global test environment set-up. 2024-03-26T21:24:29.077 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1571 ms) 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (1996 ms) 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3037 ms) 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6604 ms total) 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [----------] Global test environment tear-down 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6604 ms total) 2024-03-26T21:24:29.078 INFO:tasks.workunit.client.0.smithi088.stdout: cmd: [ PASSED ] 3 tests. 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout:i088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.238799+0000 mon.a [INF] from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-79254-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.247121+0000 mon.a [INF] from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.250690+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.255092+0000 mon.a [INF] from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.255723+0000 mon.a [INF] from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.256556+0000 mon.a [INF] from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-79584-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.260937+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:25.264308+0000 mon.a [INF] from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.037805+0000 mon.a [INF] from='client.25093 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.037863+0000 mon.a [INF] from='client.25078 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi088-79214-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.118 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.037914+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/2174745510' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.037966+0000 mon.a [INF] from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.042696+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1883281427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.077241+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.078406+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.081297+0000 mon.a [INF] from='client.24868 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.089889+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.094324+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.094529+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.098723+0000 mon.a [INF] from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.132401+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.133043+0000 mon.a [INF] from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.134223+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.134570+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.136185+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:29.119 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.136762+0000 mon.a [INF] from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.137250+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.137664+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.139190+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.139476+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.139584+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:27.140468+0000 mon.a [INF] from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041292+0000 mon.a [INF] from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-79584-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041393+0000 mon.a [INF] from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi088-79627-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041474+0000 mon.a [INF] from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi088-79254-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041551+0000 mon.a [INF] from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi088-79269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041620+0000 mon.a [INF] from='client.24868 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi088-80346-1"}]': finished 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041686+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041755+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:29.120 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041816+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041870+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.041943+0000 mon.a [INF] from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-79308-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.049882+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/3211585163' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.050581+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.052162+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.053207+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.053769+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.054791+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.055534+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.056337+0000 mon.a [INF] from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.057000+0000 mon.a [INF] from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.062201+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.066675+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.121 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.141738+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.122 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:28.183344+0000 mon.a [INF] from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.122 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.046152+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:29.122 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.046267+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:29.122 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.046336+0000 mon.a [INF] from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.122 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.046432+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:29.122 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.046542+0000 mon.a [INF] from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:29.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.052915+0000 mon.c [INF] from='client.? v1:172.21.15.8 api_io_pp: Running main() from gmock_main.cc 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: seed 79269 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (361 ms) 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (29 ms) 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (10 ms) 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (7 ms) 2024-03-26T21:24:29.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3 ms) 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (6 ms) 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-26T21:24:29.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (4 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (5 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (13 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (6 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-03-26T21:24:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (482 ms total) 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1106 ms) 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (18 ms) 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (10 ms) 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-26T21:24:29.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-03-26T21:24:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.0 deep-scrub ok 2024-03-26T21:24:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: pgmap v77: 996 pgs: 480 unknown, 64 creating+peering, 452 active+clean; 586 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 241 op/s 2024-03-26T21:24:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.3 deep-scrub starts 2024-03-26T21:24:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.3 deep-scrub ok 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.1 deep-scrub starts 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.1 deep-scrub ok 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.7 deep-scrub starts 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.7 deep-scrub ok 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.9 deep-scrub starts 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: 19.9 deep-scrub ok 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1734208084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/207756024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.25327 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3663679665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:29 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.0 deep-scrub ok 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: pgmap v77: 996 pgs: 480 unknown, 64 creating+peering, 452 active+clean; 586 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 241 op/s 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.3 deep-scrub starts 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.3 deep-scrub ok 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.1 deep-scrub starts 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.1 deep-scrub ok 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.7 deep-scrub starts 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.7 deep-scrub ok 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.9 deep-scrub starts 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: 19.9 deep-scrub ok 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:29.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1734208084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.0 deep-scrub ok 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: pgmap v77: 996 pgs: 480 unknown, 64 creating+peering, 452 active+clean; 586 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 241 op/s 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.3 deep-scrub starts 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.3 deep-scrub ok 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.1 deep-scrub starts 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.1 deep-scrub ok 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.7 deep-scrub starts 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.7 deep-scrub ok 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.9 deep-scrub starts 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: 19.9 deep-scrub ok 2024-03-26T21:24:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/149415011' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi088-79779-1"}]': finished 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi088-80549-1"}]': finished 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi088-79235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi088-80346-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1734208084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/207756024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:29.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.25327 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3663679665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/207756024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.25327 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3663679665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:29.489 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:29 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN 8:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.054377+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.058973+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/3288962253' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.072622+0000 mon.a [INF] from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.072880+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1734208084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.073055+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.075995+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/207756024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.076670+0000 mon.a [INF] from='client.35096 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.077416+0000 mon.a [INF] from='client.25327 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.077711+0000 mon.a [INF] from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.078554+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/3663679665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.086832+0000 mon.a [INF] from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.093317+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.096438+0000 mon.a [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.119729+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.132431+0000 client.admin [INF] threexx 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:29.133357+0000 mon.a [INF] from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050044+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050135+0000 mon.a [INF] from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050223+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:30.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050298+0000 mon.a [INF] from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050363+0000 mon.a [INF] from='client.25327 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050423+0000 mon.a [INF] from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050494+0000 mon.a [INF] from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.050574+0000 mon.a [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.055603+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.059507+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.064936+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.065247+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.130 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.065789+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: 19.8 deep-scrub starts 2024-03-26T21:24:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: 19.8 deep-scrub ok 2024-03-26T21:24:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: threexx 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: pgmap v80: 868 pgs: 384 unknown, 32 creating+peering, 452 active+clean; 586 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1940882810' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.25327 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:30.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:30.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: 19.8 deep-scrub starts 2024-03-26T21:24:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: 19.8 deep-scrub ok 2024-03-26T21:24:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: threexx 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: pgmap v80: 868 pgs: 384 unknown, 32 creating+peering, 452 active+clean; 586 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1940882810' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.25327 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: 19.8 deep-scrub starts 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: 19.8 deep-scrub ok 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: threexx 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: pgmap v80: 868 pgs: 384 unknown, 32 creating+peering, 452 active+clean; 586 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1940882810' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-79496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi088-79308-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-4", "overlaypool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35096 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi088-79235-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.25327 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-79214-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi088-80549-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T21:24:30.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:30.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:30.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:30 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:30.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:24:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.066128+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/947824198' entity='client.a list: Running main() from gmock_main.cc 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [----------] Global test environment set-up. 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [----------] 3 tests from NeoradosList 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [ OK ] NeoradosList.ListObjects (2560 ms) 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [ OK ] NeoradosList.ListObjectsNS (2981 ms) 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3033 ms) 2024-03-26T21:24:31.087 INFO:tasks.workunit.client.0.smithi088.stdout: list: [----------] 3 tests from NeoradosList (8574 ms total) 2024-03-26T21:24:31.088 INFO:tasks.workunit.client.0.smithi088.stdout: list: 2024-03-26T21:24:31.088 INFO:tasks.workunit.client.0.smithi088.stdout: list: [----------] Global test environment tear-down 2024-03-26T21:24:31.088 INFO:tasks.workunit.client.0.smithi088.stdout: list: [==========] 3 tests from 1 test suite ran. (8574 ms total) 2024-03-26T21:24:31.088 INFO:tasks.workunit.client.0.smithi088.stdout: list: [ PASSED ] 3 tests. 2024-03-26T21:24:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: 19.4 deep-scrub starts 2024-03-26T21:24:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: 19.4 deep-scrub ok 2024-03-26T21:24:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T21:24:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T21:24:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: fourxx 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]': finished 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:31 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: 19.4 deep-scrub starts 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: 19.4 deep-scrub ok 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: fourxx 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]': finished 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:31.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: 19.4 deep-scrub starts 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: 19.4 deep-scrub ok 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2621883844' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: fourxx 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi088-80346-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]': finished 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/947824198' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1749314028' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1083361822' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:31.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:31.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:31 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:31.588 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-26T21:24:31.588 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] Global test environment set-up. 2024-03-26T21:24:31.588 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-26T21:24:31.588 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-26T21:24:31.588 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5076 ms) 2024-03-26T21:24:31.588 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.RegisterLate (29 ms) 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: cluster: 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: id: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: health: HEALTH_WARN 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 13 pool(s) do not have an application enabled 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: services: 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.589 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: data: 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pools: 27 pools, 732 pgs 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: objects: 386 objects, 605 KiB 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: usage: 254 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pgs: 25.137% pgs not active 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 548 active+clean 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 184 creating+peering 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: io: 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: client: 2.8 KiB/s rd, 79 KiB/s wr, 71 op/s rd, 173 op/s wr 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.590 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: cluster: 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: id: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: health: HEALTH_WARN 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 16 pool(s) do not have an application enabled 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: services: 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: data: 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pools: 33 pools, 996 pgs 2024-03-26T21:24:31.591 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: objects: 238 objects, 586 KiB 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: usage: 254 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pgs: 48.193% pgs unknown 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 6.426% pgs not active 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 480 unknown 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 452 active+clean 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 64 creating+peering 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: io: 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: client: 3.0 KiB/s rd, 21 KiB/s wr, 105 op/s rd, 135 op/s wr 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: cluster: 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: id: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:24:31.592 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: health: HEALTH_WARN 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 1 cache pools are missing hit_sets 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 16 pool(s) do not have an application enabled 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: services: 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T21:24:31.593 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-26T21:24:32.069 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: api_stat: Running main() from gmock_main.cc 2024-03-26T21:24:32.069 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] Global test environment set-up. 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.Stat (363 ms) 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6 ms) 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.StatNS (18 ms) 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-26T21:24:32.070 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 5 tests from LibRadosStat (392 ms total) 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1088 ms) 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (34 ms) 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1124 ms total) 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] Global test environment tear-down 2024-03-26T21:24:32.071 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9833 ms total) 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: Running main() from gmock_main.cc 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] Global test environment set-up. 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (345 ms) 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-26T21:24:32.072 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (47 ms) 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (6 ms) 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (7 ms) 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (4 ms) 2024-03-26T21:24:32.073 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4 ms) 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3 ms) 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.TruncTest (6 ms) 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.RmXattr (14 ms) 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-03-26T21:24:32.074 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 14 tests from LibRadosIo (454 ms total) 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1084 ms) 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (31 ms) 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-26T21:24:32.075 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (4 ms) 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (11 ms) 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (3 ms) 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1159 ms total) 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] Global test environment tear-down 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9974 ms total) 2024-03-26T21:24:32.076 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ PASSED ] 24 tests. 2024-03-26T21:24:32.078 INFO:tasks.workunit.client.0.smithi088.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-26T21:24:32.078 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (7 ms) 2024-03-26T21:24:32.078 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-26T21:24:32.078 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (3 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (3 ms) 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-26T21:24:32.079 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (7 ms) 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (5 ms) 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (7 ms) 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1200 ms total) 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (9972 ms total) 2024-03-26T21:24:32.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-26T21:24:32.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-26T21:24:32.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-26T21:24:32.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-26T21:24:32.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-26T21:24:32.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: seed 79627 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (359 ms) 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (25 ms) 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3 ms) 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (20 ms) 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (408 ms total) 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-26T21:24:32.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1116 ms) 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2 ms) 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (8 ms) 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1126 ms total) 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9832 ms total) 2024-03-26T21:24:32.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-26T21:24:32.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T21:24:32.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]: dispatch 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]: dispatch 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]: dispatch 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]: dispatch 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: pgmap v83: 732 pgs: 80 creating+peering, 72 unknown, 580 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 310 op/s 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3135317478' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]': finished 2024-03-26T21:24:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]': finished 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]: dispatch 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]: dispatch 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]: dispatch 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]: dispatch 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: pgmap v83: 732 pgs: 80 creating+peering, 72 unknown, 580 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 310 op/s 2024-03-26T21:24:32.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3135317478' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]': finished 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3572332905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.35324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1690110852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3959851377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3572332905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.35324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1690110852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3959851377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.25063 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]: dispatch 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]: dispatch 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]: dispatch 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]: dispatch 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: pgmap v83: 732 pgs: 80 creating+peering, 72 unknown, 580 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 310 op/s 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3135317478' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T21:24:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi088-79627-7"}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-79584-7"}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35069 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-79254-16"}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-79269-22"}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79779-6", "pg_num": 4}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-4"}]': finished 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3572332905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.35324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1690110852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3959851377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/642972510' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:32.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:32.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:32 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]: dispatch 2024-03-26T21:24:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:24:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:24:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:24:33.148 INFO:tasks.workunit.client.0.smithi088.stdout:dmin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:33.148 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.066517+0000 mon.c [INF] from='client.? v1:172.21.15.88:0/2635229432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:33.148 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.068526+0000 mon.a [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi088-80549-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.068868+0000 mon.a [INF] from='client.34961 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.069245+0000 mon.a [INF] from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.069513+0000 mon.a [INF] from='client.35069 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-79254-16"}]: dispatch 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.069778+0000 mon.a [INF] from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-79627-7"}]: dispatch 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.069936+0000 mon.a [INF] from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-79584-7"}]: dispatch 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-03-26T21:24:30.070354+0000 mon.a [INF] from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-79269-22"}]: dispatch 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8097 ms) 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10850 ms total) 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10850 ms total) 2024-03-26T21:24:33.149 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-26T21:24:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T21:24:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.35324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1690110852' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2"}]: dispatch 2024-03-26T21:24:33.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:33 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.35324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1690110852' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2"}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:33.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.35324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi088-79214-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1690110852' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.34961 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-4", "tierpool": "test-rados-api-smithi088-79477-4-cache"}]': finished 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2"}]: dispatch 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:33 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: 19.5 deep-scrub starts 2024-03-26T21:24:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: 19.5 deep-scrub ok 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: pgmap v86: 864 pgs: 48 creating+peering, 268 unknown, 548 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 308 op/s 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]': finished 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3545619972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:34.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:34 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: 19.5 deep-scrub starts 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: 19.5 deep-scrub ok 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: pgmap v86: 864 pgs: 48 creating+peering, 268 unknown, 548 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 308 op/s 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]': finished 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]: dispatch 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3545619972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T21:24:34.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: 19.5 deep-scrub starts 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: 19.5 deep-scrub ok 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: pgmap v86: 864 pgs: 48 creating+peering, 268 unknown, 548 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 308 op/s 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "overlaypool": "test-rados-api-smithi088-79779-6"}]': finished 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi088-79355-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]: dispatch 2024-03-26T21:24:34.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3902614044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3545619972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:24:34.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:24:34.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:24:34.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:34 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: Running main() from gmock_main.cc 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [----------] Global test environment set-up. 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5588 ms) 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (7012 ms) 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12600 ms total) 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [----------] Global test environment tear-down 2024-03-26T21:24:35.069 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12600 ms total) 2024-03-26T21:24:35.070 INFO:tasks.workunit.client.0.smithi088.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-26T21:24:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]': finished 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2579960055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3331909858' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:35.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:24:35.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:24:35.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2231269957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi088-79214-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]': finished 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2579960055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3331909858' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2231269957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi088-79214-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi088-80346-2"}]': finished 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79779-6", "mode": "writeback"}]': finished 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi088-79509-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1044401140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1465855287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2579960055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3924063769' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3331909858' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:35.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2231269957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi088-79214-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:36.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-26T21:24:36.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T21:24:36.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-26T21:24:36.080 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: seed 79496 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (332 ms) 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (26 ms) 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1011 ms) 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1027 ms) 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (29 ms) 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (10 ms) 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-26T21:24:36.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2443 ms total) 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1145 ms) 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (6 ms) 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1036 ms) 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1022 ms) 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-26T21:24:36.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3 ms) 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (2 ms) 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3222 ms total) 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13900 ms total) 2024-03-26T21:24:36.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: Running main() from gmock_main.cc 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [----------] Global test environment set-up. 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ OK ] NeoRadosPools.PoolList (1523 ms) 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (1993 ms) 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2015 ms) 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-26T21:24:36.084 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4016 ms) 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2000 ms) 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2008 ms) 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [----------] 6 tests from NeoRadosPools (13556 ms total) 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [----------] Global test environment tear-down 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [==========] 6 tests from 1 test suite ran. (13556 ms total) 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: pool: [ PASSED ] 6 tests. 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: Running main() from gmock_main.cc 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] Global test environment set-up. 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-26T21:24:36.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (340 ms) 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockShared (50 ms) 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1011 ms) 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1038 ms) 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3 ms) 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.Unlock (14 ms) 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-26T21:24:36.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (13 ms) 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2 ms) 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLock (2471 ms total) 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1137 ms) 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (9 ms) 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1041 ms) 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1038 ms) 2024-03-26T21:24:36.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (4 ms) 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (4 ms) 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3240 ms total) 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] Global test environment tear-down 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13959 ms total) 2024-03-26T21:24:36.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-26T21:24:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: pgmap v89: 776 pgs: 192 creating+peering, 584 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2024-03-26T21:24:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3331909858' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2231269957' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi088-79214-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: pgmap v89: 776 pgs: 192 creating+peering, 584 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3331909858' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2231269957' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi088-79214-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:24:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: pgmap v89: 776 pgs: 192 creating+peering, 584 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi088-80549-2"}]': finished 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-79496-10"}]': finished 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3331909858' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.35135 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-79308-10"}]': finished 2024-03-26T21:24:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2231269957' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi088-79214-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4098685588' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: pool 'PoolQuotaPP_smithi088-79235-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T21:24:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:24:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]': finished 2024-03-26T21:24:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T21:24:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi088-79701-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: pool 'PoolQuotaPP_smithi088-79235-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]': finished 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi088-79701-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: pool 'PoolQuotaPP_smithi088-79235-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.35300 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi088-79355-1","app":"app1","key":"key1"}]': finished 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi088-80549-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi088-79701-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:37.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:38.190 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-26T21:24:38.190 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T21:24:38.190 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-26T21:24:38.190 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1089 ms) 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1094 ms total) 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (141 ms) 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify 2024-03-26T21:24:38.191 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3949 ms) 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5 ms) 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489846048 notify_id 360777252868 notifier_gid 35288 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489846048 notify_id 360777252865 notifier_gid 35288 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489846048 notify_id 360777252869 notifier_gid 35288 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (6 ms) 2024-03-26T21:24:38.192 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489846048 notify_id 360777252866 notifier_gid 35288 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489846048 notify_id 360777252870 notifier_gid 35288 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (5 ms) 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489846048 notify_id 360777252867 notifier_gid 35288 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (4 ms) 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: trying... 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489871312 notify_id 360777252871 notifier_gid 35288 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: timed out 2024-03-26T21:24:38.193 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: pgmap v92: 552 pgs: 64 unknown, 488 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1599655556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2653332439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1408475445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1771920472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.35429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.35435 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: pgmap v92: 552 pgs: 64 unknown, 488 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1599655556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2653332439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1408475445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1771920472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.35429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.35435 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:38.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: pgmap v92: 552 pgs: 64 unknown, 488 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-03-26T21:24:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T21:24:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1599655556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2653332439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1408475445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1771920472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.35429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.35435 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:39.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi088-79701-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.35429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.35435 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:40.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:40.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:40.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:39 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi088-79701-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.35429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.35435 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi088-79701-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi088-80549-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.35429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi088-79544-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.35435 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi088-79214-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:40.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:39 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:40.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:24:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:24:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[39202]: pgmap v95: 792 pgs: 304 unknown, 488 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[39202]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T21:24:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:41.138 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[30075]: pgmap v95: 792 pgs: 304 unknown, 488 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:41.138 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:41.138 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[30075]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T21:24:41.138 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:41.138 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:41.138 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:40 smithi138 ceph-mon[33706]: pgmap v95: 792 pgs: 304 unknown, 488 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:40 smithi138 ceph-mon[33706]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T21:24:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3320953497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3142732905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.38519 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.28666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T21:24:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3320953497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3142732905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.38519 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.28666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3320953497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3142732905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.38519 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.28666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:42.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:24:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:24:42.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_watc api_watch_notify: Running main() from gmock_main.cc 2024-03-26T21:24:42.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-26T21:24:42.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-26T21:24:42.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (369 ms) 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94731518028048 err -107 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1008 ms) 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94731519430640 err -107 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1033 ms) 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 322122547200 cookie 94731519435104 2024-03-26T21:24:42.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (48 ms) 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 322122547201 cookie 94731519436368 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (10 ms) 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 322122547201 cookie 94731519457808 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (5 ms) 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 322122547202 cookie 94731519461776 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 322122547202 cookie 94731519480080 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (6 ms) 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 322122547202 cookie 94731519461776 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 326417514499 cookie 94731519461776 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3005 ms) 2024-03-26T21:24:42.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94731519461648 err -107 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 24748 notify_id 356482285570 cookie 94731519461648 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5008 ms) 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94731519480080 err -107 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11497 ms total) 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T21:24:42.850 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1125 ms) 2024-03-26T21:24:42.851 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1125 ms total) 2024-03-26T21:24:42.851 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: 2024-03-26T21:24:42.851 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-26T21:24:42.851 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20587 ms total) 2024-03-26T21:24:42.851 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-26T21:24:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: pgmap v98: 752 pgs: 160 unknown, 592 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 3 op/s 2024-03-26T21:24:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]': finished 2024-03-26T21:24:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.38519 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.28666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2306782229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.31054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:43.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:43.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: pgmap v98: 752 pgs: 160 unknown, 592 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 3 op/s 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]': finished 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.38519 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.28666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:24:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2306782229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.31054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: pgmap v98: 752 pgs: 160 unknown, 592 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 3 op/s 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:43.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-7"}]': finished 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.38519 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi088-79214-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.28666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-80036-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]: dispatch 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1027084939' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi088-79701-12"}]': finished 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2306782229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:43.239 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.31054 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:43.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:43.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]: dispatch 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.31054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]': finished 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2155712664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.32101 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:43 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.31054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]': finished 2024-03-26T21:24:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T21:24:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2155712664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.32101 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.31054 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-7"}]': finished 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2155712664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3884209421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.32101 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:44.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:43 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]: dispatch 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: Running main() from gmock_main.cc 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [----------] Global test environment set-up. 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6532 ms) 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7024 ms) 2024-03-26T21:24:44.865 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-26T21:24:44.866 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8781 ms) 2024-03-26T21:24:44.866 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [----------] 3 tests from NeoradosECList (22337 ms total) 2024-03-26T21:24:44.866 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: 2024-03-26T21:24:44.866 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [----------] Global test environment tear-down 2024-03-26T21:24:44.866 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (22337 ms total) 2024-03-26T21:24:44.866 INFO:tasks.workunit.client.0.smithi088.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-26T21:24:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: pgmap v101: 648 pgs: 96 unknown, 552 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:24:45.244 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:45.244 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:45.244 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: from='client.32101 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:45.244 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:45.244 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T21:24:45.244 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/532508366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:45.244 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:44 smithi138 ceph-mon[33706]: from='client.33181 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: pgmap v101: 648 pgs: 96 unknown, 552 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: from='client.32101 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/532508366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[30075]: from='client.33181 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: pgmap v101: 648 pgs: 96 unknown, 552 active+clean; 145 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:45.245 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:45.246 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: from='client.32101 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi088-79214-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:45.246 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: from='client.35402 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi088-80549-3"}]': finished 2024-03-26T21:24:45.246 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T21:24:45.246 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/532508366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:45.246 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:44 smithi088 ceph-mon[39202]: from='client.33181 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:45.928 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-26T21:24:45.928 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-26T21:24:45.928 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-26T21:24:45.928 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2716 ms) 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (2979 ms) 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3013 ms) 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3003 ms) 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2695 ms) 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-26T21:24:45.929 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3014 ms) 2024-03-26T21:24:45.930 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-26T21:24:45.930 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (6147 ms) 2024-03-26T21:24:45.930 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (23567 ms total) 2024-03-26T21:24:45.930 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: 2024-03-26T21:24:45.930 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-26T21:24:45.930 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (23568 ms total) 2024-03-26T21:24:45.930 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-26T21:24:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:45 smithi138 ceph-mon[33706]: from='client.33181 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:45 smithi138 ceph-mon[33706]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T21:24:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:45 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[30075]: from='client.33181 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[30075]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[39202]: from='client.33181 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[39202]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:45 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: pgmap v104: 744 pgs: 96 creating+peering, 648 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T21:24:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T21:24:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2389803205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: from='client.35371 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: pgmap v104: 744 pgs: 96 creating+peering, 648 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T21:24:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2389803205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: from='client.35371 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: pgmap v104: 744 pgs: 96 creating+peering, 648 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T21:24:47.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2389803205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:47.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: from='client.35371 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:47.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:47.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:47.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]: dispatch 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]: dispatch 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.35371 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]': finished 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3402008085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3620209706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.45419 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.36415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.35371 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]': finished 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3402008085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3620209706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.45419 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.36415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]: dispatch 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]: dispatch 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.35371 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi088-79214-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4"}]': finished 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3402008085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3620209706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:48.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]: dispatch 2024-03-26T21:24:48.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.45419 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.36415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:48.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: pgmap v107: 616 pgs: 128 unknown, 488 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-03-26T21:24:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]': finished 2024-03-26T21:24:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]': finished 2024-03-26T21:24:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.45419 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.36415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: pgmap v107: 616 pgs: 128 unknown, 488 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]': finished 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]': finished 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.45419 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.36415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]: dispatch 2024-03-26T21:24:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: pgmap v107: 616 pgs: 128 unknown, 488 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-9"}]': finished 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-79779-4", "tierpool": "test-rados-api-smithi088-79779-6"}]': finished 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.45419 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.36415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi088-79235-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]: dispatch 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/298428802' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]: dispatch 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:49.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:49.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:24:49.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]': finished 2024-03-26T21:24:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:24:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T21:24:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:50.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:50.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2075918116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:50.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.47699 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:50.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:50.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:50.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]': finished 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2075918116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.47699 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-9", "mode": "writeback"}]': finished 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi088-79779-6", "pool2": "test-rados-api-smithi088-79779-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2075918116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.47699 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:24:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:50.781 INFO:tasks.workunit.client.0.smithi088.stdout:h_notify_pp: flushed 2024-03-26T21:24:50.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3005 ms) 2024-03-26T21:24:50.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-26T21:24:50.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: trying... 2024-03-26T21:24:50.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489873168 notify_id 373662154752 notifier_gid 35288 2024-03-26T21:24:50.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: timed out 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushed 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3024 ms) 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: List watches 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489873168 notify_id 386547056641 notifier_gid 35288 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 done 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: watch_check 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: unwatch2 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: done 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3034 ms) 2024-03-26T21:24:50.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: List watches 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 93986489873168 notify_id 399431958536 notifier_gid 35288 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 done 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: watch_check 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: unwatch2 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: done 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3020 ms) 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16213 ms total) 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28479 ms total) 2024-03-26T21:24:50.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-26T21:24:50.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:24:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:24:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: pgmap v110: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: from='client.47699 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:24:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T21:24:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:51.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1741913516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:51.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: from='client.39658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:51.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: pgmap v110: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: from='client.47699 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1741913516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: from='client.39658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: pgmap v110: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: from='client.47699 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi088-79214-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:24:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T21:24:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]: dispatch 2024-03-26T21:24:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1741913516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: from='client.39658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: laundry: 1 daemon active (1 hosts) 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: data: 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pools: 32 pools, 868 pgs 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: objects: 238 objects, 586 KiB 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: usage: 254 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pgs: 44.240% pgs unknown 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 3.687% pgs not active 2024-03-26T21:24:51.659 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 452 active+clean 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 384 unknown 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 32 creating+peering 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4163 ms) 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.Status (20038 ms) 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] 4 tests from LibRadosService (29306 ms total) 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] Global test environment tear-down 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29306 ms total) 2024-03-26T21:24:51.660 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ PASSED ] 4 tests. 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:24:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]': finished 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[39202]: from='client.39658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[39202]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T21:24:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:24:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]': finished 2024-03-26T21:24:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[30075]: from='client.39658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[30075]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T21:24:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:24:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:52 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:24:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:52 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi088-79509-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:24:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:52 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-9"}]': finished 2024-03-26T21:24:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:52 smithi138 ceph-mon[33706]: from='client.39658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi088-79235-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:52 smithi138 ceph-mon[33706]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T21:24:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:52.663 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5073 ms) 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (49 ms) 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20034 ms) 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: attempt 0 of 20 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5150 ms) 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30306 ms total) 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: 2024-03-26T21:24:52.664 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-26T21:24:52.665 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30307 ms total) 2024-03-26T21:24:52.665 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: pgmap v113: 644 pgs: 96 creating+peering, 64 unknown, 484 active+clean; 145 MiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1483337008' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/50594601' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: from='client.41965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: pgmap v113: 644 pgs: 96 creating+peering, 64 unknown, 484 active+clean; 145 MiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1483337008' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/50594601' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: from='client.41965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: pgmap v113: 644 pgs: 96 creating+peering, 64 unknown, 484 active+clean; 145 MiB data, 753 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:24:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1483337008' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T21:24:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T21:24:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T21:24:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/50594601' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: from='client.41965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:54.812 INFO:tasks.workunit.client.0.smithi088.stdout: misc: Running main() from gmock_main.cc 2024-03-26T21:24:54.812 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-26T21:24:54.812 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [----------] Global test environment set-up. 2024-03-26T21:24:54.812 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-26T21:24:54.812 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-26T21:24:54.812 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.Version (1527 ms) 2024-03-26T21:24:54.812 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (1998 ms) 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.LongName (3029 ms) 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2995 ms) 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3001 ms) 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2692 ms) 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.Exec (3065 ms) 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3040 ms) 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-26T21:24:54.813 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2903 ms) 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3002 ms) 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1999 ms) 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3021 ms) 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [----------] 12 tests from NeoRadosMisc (32273 ms total) 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [----------] Global test environment tear-down 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [==========] 12 tests from 1 test suite ran. (32273 ms total) 2024-03-26T21:24:54.814 INFO:tasks.workunit.client.0.smithi088.stdout: misc: [ PASSED ] 12 tests. 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: pgmap v116: 588 pgs: 168 unknown, 420 active+clean; 145 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='client.41965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/813049787' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='client.52118 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:54 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: pgmap v116: 588 pgs: 168 unknown, 420 active+clean; 145 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='client.41965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/813049787' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='client.52118 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: pgmap v116: 588 pgs: 168 unknown, 420 active+clean; 145 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='client.41965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi088-79214-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79505-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/813049787' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='client.52118 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:24:55.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:54 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:24:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.52118 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T21:24:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T21:24:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T21:24:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]': finished 2024-03-26T21:24:56.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T21:24:56.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/963350866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:55 smithi138 ceph-mon[33706]: from='client.45511 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.52118 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]': finished 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/963350866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[30075]: from='client.45511 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.52118 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi088-79235-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache", "force_nonempty":""}]': finished 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/963350866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:55 smithi088 ceph-mon[39202]: from='client.45511 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:56.754 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] Global test environment set-up. 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-03-26T21:24:22.254+0000 7fef02235780 -1 failed for service _ceph-mon._tcp 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-03-26T21:24:22.254+0000 7fef02235780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (92 ms) 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-26T21:24:56.755 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2015 ms) 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2107 ms total) 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x55bf13ce9660 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x55bf140e3200 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 2 aios 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x55bf13ce9660 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x55bf140e3200 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: done. 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5797 ms) 2024-03-26T21:24:56.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5797 ms total) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.Exec (79 ms) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (8 ms) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4572 ms) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-26T21:24:56.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-26T21:24:56.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (19841 ms) 2024-03-26T21:24:56.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 8 tests from LibRadosMisc (24503 ms total) 2024-03-26T21:24:56.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-03-26T21:24:56.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] Global test environment tear-down 2024-03-26T21:24:56.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (34586 ms total) 2024-03-26T21:24:56.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: pgmap v119: 556 pgs: 32 creating+peering, 524 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.45511 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T21:24:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:24:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3407601257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:24:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]: dispatch 2024-03-26T21:24:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.55517 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:56 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]: dispatch 2024-03-26T21:24:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: pgmap v119: 556 pgs: 32 creating+peering, 524 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.45511 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3407601257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.55517 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]: dispatch 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: pgmap v119: 556 pgs: 32 creating+peering, 524 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79505-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.45511 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi088-79214-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T21:24:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:24:57.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3407601257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:57.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:24:57.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4244156161' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]: dispatch 2024-03-26T21:24:57.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.55517 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:57.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:56 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]: dispatch 2024-03-26T21:24:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: 19.6 deep-scrub starts 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: 19.6 deep-scrub ok 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: 19.1 deep-scrub starts 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: 19.1 deep-scrub ok 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]': finished 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.55517 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]': finished 2024-03-26T21:24:58.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T21:24:58.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]: dispatch 2024-03-26T21:24:58.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]: dispatch 2024-03-26T21:24:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: 19.6 deep-scrub starts 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: 19.6 deep-scrub ok 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: 19.1 deep-scrub starts 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: 19.1 deep-scrub ok 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]': finished 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.55517 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]': finished 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: 19.6 deep-scrub starts 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: 19.6 deep-scrub ok 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: 19.1 deep-scrub starts 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: 19.1 deep-scrub ok 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-11"}]': finished 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.55517 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi088-79235-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.41956 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79505-10", "tierpool":"test-rados-api-smithi088-79505-10-cache"}]': finished 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]: dispatch 2024-03-26T21:24:58.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]: dispatch 2024-03-26T21:24:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: pgmap v122: 620 pgs: 128 unknown, 32 creating+peering, 460 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 1 op/s 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.2 deep-scrub starts 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.2 deep-scrub ok 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.3 deep-scrub starts 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.3 deep-scrub ok 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.7 deep-scrub starts 2024-03-26T21:24:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.7 deep-scrub ok 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.0 deep-scrub starts 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.0 deep-scrub ok 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.5 deep-scrub starts 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.5 deep-scrub ok 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.8 deep-scrub starts 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: 19.8 deep-scrub ok 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]': finished 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2369303040' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:59.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:58 smithi138 ceph-mon[33706]: from='client.56549 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: pgmap v122: 620 pgs: 128 unknown, 32 creating+peering, 460 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 1 op/s 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.2 deep-scrub starts 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.2 deep-scrub ok 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.3 deep-scrub starts 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.3 deep-scrub ok 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.7 deep-scrub starts 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.7 deep-scrub ok 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.0 deep-scrub starts 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.0 deep-scrub ok 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.5 deep-scrub starts 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.5 deep-scrub ok 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.8 deep-scrub starts 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: 19.8 deep-scrub ok 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]': finished 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2369303040' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[30075]: from='client.56549 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T21:24:59.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.138:0/3290033355' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: pgmap v122: 620 pgs: 128 unknown, 32 creating+peering, 460 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 1 op/s 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.2 deep-scrub starts 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.2 deep-scrub ok 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.3 deep-scrub starts 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.3 deep-scrub ok 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.7 deep-scrub starts 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.7 deep-scrub ok 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.0 deep-scrub starts 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.0 deep-scrub ok 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.5 deep-scrub starts 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.5 deep-scrub ok 2024-03-26T21:24:59.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.8 deep-scrub starts 2024-03-26T21:24:59.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: 19.8 deep-scrub ok 2024-03-26T21:24:59.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:24:59.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-11", "mode": "writeback"}]': finished 2024-03-26T21:24:59.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T21:24:59.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2369303040' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:59.238 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:58 smithi088 ceph-mon[39202]: from='client.56549 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:24:59.746 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: Running main() from gmock_main.cc 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] Global test environment set-up. 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2861 ms) 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3022 ms) 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3003 ms) 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3009 ms) 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2735 ms) 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-26T21:24:59.747 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5057 ms) 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4888 ms) 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5043 ms) 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7948 ms) 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] 9 tests from LibRadosPools (37566 ms total) 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] Global test environment tear-down 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37566 ms total) 2024-03-26T21:24:59.748 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: 19.9 deep-scrub starts 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: 19.9 deep-scrub ok 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: 19.4 deep-scrub starts 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: 19.4 deep-scrub ok 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: from='client.56549 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/332316306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: from='client.47689 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:00.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:00.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:24:59 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: 19.9 deep-scrub starts 2024-03-26T21:25:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: 19.9 deep-scrub ok 2024-03-26T21:25:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: 19.4 deep-scrub starts 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: 19.4 deep-scrub ok 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: from='client.56549 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/332316306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: from='client.47689 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:00.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: 19.9 deep-scrub starts 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: 19.9 deep-scrub ok 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: 19.4 deep-scrub starts 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: 19.4 deep-scrub ok 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: from='client.56549 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi088-79214-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/332316306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: from='client.47689 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:00.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:24:59 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:00.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:25:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-mon[33706]: pgmap v125: 556 pgs: 96 unknown, 460 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-mon[33706]: from='client.47689 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-mon[33706]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T21:25:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:25:01.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:25:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[30075]: pgmap v125: 556 pgs: 96 unknown, 460 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[30075]: from='client.47689 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[30075]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T21:25:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[39202]: pgmap v125: 556 pgs: 96 unknown, 460 active+clean; 146 MiB data, 763 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[39202]: from='client.47689 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[39202]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:25:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]: dispatch 2024-03-26T21:25:02.143 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:02.143 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:02.143 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:02.143 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]': finished 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/580586378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:02.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:01 smithi138 ceph-mon[33706]: from='client.56570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]': finished 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/580586378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[39202]: from='client.56570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-11"}]': finished 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/580586378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:01 smithi088 ceph-mon[30075]: from='client.56570 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:25:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:25:03.197 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: pgmap v128: 460 pgs: 32 creating+peering, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 8 op/s 2024-03-26T21:25:03.197 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:03.197 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:25:03.197 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: from='client.56570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:03.197 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T21:25:03.197 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:03.197 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:03.198 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1399217832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:03.198 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:02 smithi138 ceph-mon[33706]: from='client.56576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: pgmap v128: 460 pgs: 32 creating+peering, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 8 op/s 2024-03-26T21:25:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:25:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: from='client.56570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T21:25:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1399217832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[30075]: from='client.56576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: pgmap v128: 460 pgs: 32 creating+peering, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 8 op/s 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: from='client.56570 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi088-79214-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1276463223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1399217832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:02 smithi088 ceph-mon[39202]: from='client.56576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:03.759 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: Running main() from gmock_main.cc 2024-03-26T21:25:03.759 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-26T21:25:03.759 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [----------] Global test environment set-up. 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2523 ms) 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (2988 ms) 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3018 ms) 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3008 ms) 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2678 ms) 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3008 ms) 2024-03-26T21:25:03.760 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3081 ms) 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3022 ms) 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (2896 ms) 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3015 ms) 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3017 ms) 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2954 ms) 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3001 ms) 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-26T21:25:03.761 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3011 ms) 2024-03-26T21:25:03.762 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41221 ms total) 2024-03-26T21:25:03.762 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: 2024-03-26T21:25:03.762 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [----------] Global test environment tear-down 2024-03-26T21:25:03.762 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41221 ms total) 2024-03-26T21:25:03.762 INFO:tasks.workunit.client.0.smithi088.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-26T21:25:03.770 INFO:tasks.workunit.client.0.smithi088.stdout: io: Running main() from gmock_main.cc 2024-03-26T21:25:03.770 INFO:tasks.workunit.client.0.smithi088.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-26T21:25:03.770 INFO:tasks.workunit.client.0.smithi088.stdout: io: [----------] Global test environment set-up. 2024-03-26T21:25:03.770 INFO:tasks.workunit.client.0.smithi088.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-26T21:25:03.770 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.Limits (2557 ms) 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3016 ms) 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.ReadOp (3040 ms) 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.SparseRead (2981 ms) 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2667 ms) 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3029 ms) 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3098 ms) 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-26T21:25:03.771 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3024 ms) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2857 ms) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.Trunc (3033 ms) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.Remove (2992 ms) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2971 ms) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.RmXattr (3007 ms) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3013 ms) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: [----------] 14 tests from NeoRadosIo (41286 ms total) 2024-03-26T21:25:03.772 INFO:tasks.workunit.client.0.smithi088.stdout: io: 2024-03-26T21:25:03.773 INFO:tasks.workunit.client.0.smithi088.stdout: io: [----------] Global test environment tear-down 2024-03-26T21:25:03.773 INFO:tasks.workunit.client.0.smithi088.stdout: io: [==========] 14 tests from 1 test suite ran. (41286 ms total) 2024-03-26T21:25:03.773 INFO:tasks.workunit.client.0.smithi088.stdout: io: [ PASSED ] 14 tests. 2024-03-26T21:25:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:25:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.56576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:04.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:03 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:04.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:04.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.56576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:04.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.38644 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi088-79509-10"}]': finished 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.56576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi088-79235-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:04.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:03 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: pgmap v131: 516 pgs: 128 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1568653556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:04 smithi138 ceph-mon[33706]: from='client.47716 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: pgmap v131: 516 pgs: 128 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-26T21:25:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1568653556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[30075]: from='client.47716 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: pgmap v131: 516 pgs: 128 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1568653556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:04 smithi088 ceph-mon[39202]: from='client.47716 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:05 smithi138 ceph-mon[33706]: from='client.47716 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:05 smithi138 ceph-mon[33706]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T21:25:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1986521181' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:05 smithi138 ceph-mon[33706]: from='client.47725 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[39202]: from='client.47716 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[39202]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T21:25:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1986521181' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[39202]: from='client.47725 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[30075]: from='client.47716 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi088-79214-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[30075]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T21:25:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1986521181' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:05 smithi088 ceph-mon[30075]: from='client.47725 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:06 smithi138 ceph-mon[33706]: pgmap v134: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s wr, 22 op/s 2024-03-26T21:25:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:06 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:06 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:06 smithi138 ceph-mon[33706]: from='client.47725 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:06 smithi138 ceph-mon[33706]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T21:25:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[30075]: pgmap v134: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s wr, 22 op/s 2024-03-26T21:25:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[30075]: from='client.47725 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[30075]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[39202]: pgmap v134: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s wr, 22 op/s 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi088-79509-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[39202]: from='client.47725 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi088-79235-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:06 smithi088 ceph-mon[39202]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T21:25:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:07 smithi138 ceph-mon[33706]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T21:25:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2330520993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:07 smithi138 ceph-mon[33706]: from='client.47728 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[39202]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T21:25:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2330520993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[39202]: from='client.47728 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[30075]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T21:25:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2330520993' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:07 smithi088 ceph-mon[30075]: from='client.47728 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:08 smithi138 ceph-mon[33706]: pgmap v137: 460 pgs: 40 unknown, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-26T21:25:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:08 smithi138 ceph-mon[33706]: from='client.47728 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:08 smithi138 ceph-mon[33706]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T21:25:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3776134255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:08 smithi138 ceph-mon[33706]: from='client.47734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[39202]: pgmap v137: 460 pgs: 40 unknown, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-26T21:25:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[39202]: from='client.47728 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[39202]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T21:25:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3776134255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[39202]: from='client.47734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[30075]: pgmap v137: 460 pgs: 40 unknown, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-03-26T21:25:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[30075]: from='client.47728 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi088-79214-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[30075]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T21:25:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3776134255' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:08 smithi088 ceph-mon[30075]: from='client.47734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='client.47734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:25:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T21:25:10.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='client.47734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='client.47734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi088-79235-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:25:10.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:10.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T21:25:10.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:10.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: pgmap v140: 492 pgs: 72 unknown, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]': finished 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]: dispatch 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]: dispatch 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1953984141' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[39202]: from='client.47740 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:11.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: pgmap v140: 492 pgs: 72 unknown, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]': finished 2024-03-26T21:25:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T21:25:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]: dispatch 2024-03-26T21:25:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]: dispatch 2024-03-26T21:25:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1953984141' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:10 smithi088 ceph-mon[30075]: from='client.47740 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: pgmap v140: 492 pgs: 72 unknown, 420 active+clean; 217 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-13"}]': finished 2024-03-26T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]: dispatch 2024-03-26T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]: dispatch 2024-03-26T21:25:11.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1953984141' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:11.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:10 smithi138 ceph-mon[33706]: from='client.47740 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]': finished 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: from='client.47740 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1569722328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:11 smithi138 ceph-mon[33706]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:25:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]': finished 2024-03-26T21:25:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: from='client.47740 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1569722328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[30075]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-13", "mode": "writeback"}]': finished 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: from='client.47740 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi088-79214-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T21:25:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1569722328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:11 smithi088 ceph-mon[39202]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:25:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: pgmap v143: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-26T21:25:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: 159.0 scrub starts 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: 159.0 scrub ok 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:12 smithi138 ceph-mon[33706]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T21:25:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: pgmap v143: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-26T21:25:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: 159.0 scrub starts 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: 159.0 scrub ok 2024-03-26T21:25:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[39202]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: pgmap v143: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-26T21:25:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-26T21:25:13.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: 159.0 scrub starts 2024-03-26T21:25:13.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: 159.0 scrub ok 2024-03-26T21:25:13.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi088-79235-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:13.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:12 smithi088 ceph-mon[30075]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T21:25:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-26T21:25:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-26T21:25:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-26T21:25:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-26T21:25:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.4 scrub starts 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.4 scrub ok 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.3 scrub starts 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.3 scrub ok 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.5 scrub starts 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.5 scrub ok 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.2 scrub starts 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: 159.2 scrub ok 2024-03-26T21:25:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T21:25:14.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2519930742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi088-79214-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.4 scrub starts 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.4 scrub ok 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.3 scrub starts 2024-03-26T21:25:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.3 scrub ok 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.5 scrub starts 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.5 scrub ok 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.2 scrub starts 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: 159.2 scrub ok 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2519930742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi088-79214-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-03-26T21:25:14.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.4 scrub starts 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.4 scrub ok 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.3 scrub starts 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.3 scrub ok 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.5 scrub starts 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.5 scrub ok 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.2 scrub starts 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: 159.2 scrub ok 2024-03-26T21:25:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T21:25:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2519930742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi088-79214-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: 159.8 scrub starts 2024-03-26T21:25:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: 159.8 scrub ok 2024-03-26T21:25:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: 159.1 scrub starts 2024-03-26T21:25:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: 159.1 scrub ok 2024-03-26T21:25:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: pgmap v146: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-26T21:25:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2519930742' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi088-79214-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T21:25:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1238298649' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:14 smithi138 ceph-mon[33706]: from='client.56621 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: 159.8 scrub starts 2024-03-26T21:25:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: 159.8 scrub ok 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: 159.1 scrub starts 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: 159.1 scrub ok 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: pgmap v146: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2519930742' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi088-79214-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1238298649' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[30075]: from='client.56621 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: 159.8 scrub starts 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: 159.8 scrub ok 2024-03-26T21:25:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: 159.1 scrub starts 2024-03-26T21:25:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: 159.1 scrub ok 2024-03-26T21:25:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: pgmap v146: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 101 op/s 2024-03-26T21:25:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2519930742' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi088-79214-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T21:25:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1238298649' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:14 smithi088 ceph-mon[39202]: from='client.56621 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:15 smithi138 ceph-mon[33706]: from='client.56621 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:15 smithi138 ceph-mon[33706]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T21:25:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:15 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[30075]: from='client.56621 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[30075]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T21:25:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[39202]: from='client.56621 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi088-79235-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[39202]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T21:25:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:15 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: pgmap v149: 492 pgs: 32 creating+peering, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.5 KiB/s wr, 78 op/s 2024-03-26T21:25:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T21:25:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:17.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3484108804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:17.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:16 smithi138 ceph-mon[33706]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: pgmap v149: 492 pgs: 32 creating+peering, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.5 KiB/s wr, 78 op/s 2024-03-26T21:25:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3484108804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[30075]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: pgmap v149: 492 pgs: 32 creating+peering, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.5 KiB/s wr, 78 op/s 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2742896463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]: dispatch 2024-03-26T21:25:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3484108804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:17.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:16 smithi088 ceph-mon[39202]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:17.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-26T21:25:17.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-26T21:25:17.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-26T21:25:17.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-26T21:25:17.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2010 ms) 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2010 ms) 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2014 ms) 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2008 ms) 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8042 ms total) 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (3720 ms) 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4191 ms) 2024-03-26T21:25:17.782 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4909 ms) 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (12821 ms total) 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3002 ms) 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1970 ms) 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1999 ms) 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2006 ms) 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8977 ms total) 2024-03-26T21:25:17.783 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4038 ms) 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4004 ms) 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8042 ms total) 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (55600 ms total) 2024-03-26T21:25:17.784 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2129753852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/649524549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:17 smithi138 ceph-mon[33706]: from='client.56636 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T21:25:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2129753852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/649524549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[39202]: from='client.56636 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi088-79509-15"}]': finished 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi088-79214-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2129753852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/649524549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:17 smithi088 ceph-mon[30075]: from='client.56636 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:18 smithi138 ceph-mon[33706]: pgmap v152: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-26T21:25:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:18 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:18 smithi138 ceph-mon[33706]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:18 smithi138 ceph-mon[33706]: from='client.56636 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:18 smithi138 ceph-mon[33706]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T21:25:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[30075]: pgmap v152: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-26T21:25:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[30075]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[30075]: from='client.56636 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[30075]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T21:25:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[39202]: pgmap v152: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-26T21:25:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[39202]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79544-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[39202]: from='client.56636 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi088-79235-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:18 smithi088 ceph-mon[39202]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T21:25:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:19 smithi138 ceph-mon[33706]: 159.9 scrub starts 2024-03-26T21:25:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:19 smithi138 ceph-mon[33706]: 159.9 scrub ok 2024-03-26T21:25:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:19 smithi138 ceph-mon[33706]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T21:25:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4137211586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79214-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[30075]: 159.9 scrub starts 2024-03-26T21:25:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[30075]: 159.9 scrub ok 2024-03-26T21:25:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[30075]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T21:25:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4137211586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79214-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[39202]: 159.9 scrub starts 2024-03-26T21:25:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[39202]: 159.9 scrub ok 2024-03-26T21:25:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[39202]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T21:25:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4137211586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79214-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:20.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: pgmap v155: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: 159.6 scrub starts 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: 159.6 scrub ok 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4137211586' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79214-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1157118271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:20 smithi138 ceph-mon[33706]: from='client.47782 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: pgmap v155: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: 159.6 scrub starts 2024-03-26T21:25:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: 159.6 scrub ok 2024-03-26T21:25:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4137211586' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79214-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1157118271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[30075]: from='client.47782 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: pgmap v155: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: 159.6 scrub starts 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: 159.6 scrub ok 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4137211586' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79214-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1157118271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:20 smithi088 ceph-mon[39202]: from='client.47782 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: 159.7 scrub starts 2024-03-26T21:25:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: 159.7 scrub ok 2024-03-26T21:25:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.47782 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T21:25:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:22.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:21 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: 159.7 scrub starts 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: 159.7 scrub ok 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.47782 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: 159.7 scrub starts 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: 159.7 scrub ok 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.47782 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi088-79235-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T21:25:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:21 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:25:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:25:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: pgmap v158: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T21:25:23.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:23.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1687996666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:23.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:23.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:22 smithi138 ceph-mon[33706]: from='client.56675 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: pgmap v158: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1687996666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[39202]: from='client.56675 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: pgmap v158: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1687996666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:22 smithi088 ceph-mon[30075]: from='client.56675 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:23 smithi138 ceph-mon[33706]: from='client.56675 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:23 smithi138 ceph-mon[33706]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T21:25:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3906571153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:23 smithi138 ceph-mon[33706]: from='client.56690 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[39202]: from='client.56675 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[39202]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T21:25:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3906571153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:24.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[39202]: from='client.56690 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:24.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:24.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[30075]: from='client.56675 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi088-79214-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:24.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[30075]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T21:25:24.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3906571153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:24.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:23 smithi088 ceph-mon[30075]: from='client.56690 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:24 smithi138 ceph-mon[33706]: pgmap v161: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:24 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:24 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:24 smithi138 ceph-mon[33706]: from='client.56690 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:25.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:24 smithi138 ceph-mon[33706]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T21:25:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[39202]: pgmap v161: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[39202]: from='client.56690 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[39202]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[30075]: pgmap v161: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi088-79544-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[30075]: from='client.56690 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:24 smithi088 ceph-mon[30075]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T21:25:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:26.239 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:25 smithi138 ceph-mon[33706]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T21:25:26.239 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/262764192' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:26.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:25 smithi138 ceph-mon[33706]: from='client.47821 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:26.240 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:26.240 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[30075]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T21:25:26.240 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/262764192' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:26.240 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[30075]: from='client.47821 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:26.240 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:26.241 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[39202]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T21:25:26.241 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/262764192' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:26.241 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:25 smithi088 ceph-mon[39202]: from='client.47821 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:26 smithi138 ceph-mon[33706]: pgmap v164: 428 pgs: 8 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:25:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:26 smithi138 ceph-mon[33706]: from='client.47821 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:26 smithi138 ceph-mon[33706]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T21:25:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3404016488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:26 smithi138 ceph-mon[33706]: from='client.56702 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[30075]: pgmap v164: 428 pgs: 8 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:25:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[30075]: from='client.47821 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[30075]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3404016488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[30075]: from='client.56702 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[39202]: pgmap v164: 428 pgs: 8 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[39202]: from='client.47821 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi088-79214-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[39202]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3404016488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:26 smithi088 ceph-mon[39202]: from='client.56702 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.56702 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T21:25:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2"}]: dispatch 2024-03-26T21:25:28.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:27 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:28.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T21:25:28.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.56702 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2"}]: dispatch 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T21:25:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.56702 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi088-79235-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T21:25:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2"}]: dispatch 2024-03-26T21:25:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:27 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: pgmap v167: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:25:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3396636622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:28 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: pgmap v167: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:25:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3396636622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: pgmap v167: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi088-79537-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T21:25:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3396636622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: from='client.56708 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:28 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: seed 79544 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2007 ms) 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2007 ms) 2024-03-26T21:25:29.795 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2017 ms) 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2007 ms) 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4009 ms) 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (12047 ms total) 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4214 ms) 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3763 ms) 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6026 ms) 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-26T21:25:29.796 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3964 ms) 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2051 ms) 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2484-g42fc52b8/rpm/el9/BUILD/ceph-19.0.0-2484-g42fc52b8/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi088-79544-7 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17965 ms) 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (37983 ms total) 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-03-26T21:25:29.797 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-26T21:25:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5054 ms) 2024-03-26T21:25:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5062 ms total) 2024-03-26T21:25:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-03-26T21:25:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-26T21:25:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-26T21:25:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3004 ms) 2024-03-26T21:25:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/418688045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:29 smithi138 ceph-mon[33706]: from='client.47839 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/418688045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[39202]: from='client.47839 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: from='client.56708 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi088-79214-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T21:25:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T21:25:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:25:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T21:25:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/418688045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:29 smithi088 ceph-mon[30075]: from='client.47839 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:30.955 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:25:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-mon[33706]: pgmap v170: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T21:25:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-mon[33706]: from='client.47839 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-mon[33706]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T21:25:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:25:31.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:30 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:25:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[30075]: pgmap v170: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T21:25:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[30075]: from='client.47839 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[30075]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[39202]: pgmap v170: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[39202]: from='client.47839 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi088-79235-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[39202]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2439327517' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:25:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:30 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T21:25:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:31 smithi138 ceph-mon[33706]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]': finished 2024-03-26T21:25:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:31 smithi138 ceph-mon[33706]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T21:25:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1548175921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:31 smithi138 ceph-mon[33706]: from='client.47845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[30075]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]': finished 2024-03-26T21:25:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[30075]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T21:25:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1548175921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[30075]: from='client.47845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[39202]: from='client.24730 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi088-79537-1","app":"app1","key":"key1"}]': finished 2024-03-26T21:25:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[39202]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T21:25:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1548175921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:31 smithi088 ceph-mon[39202]: from='client.47845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:25:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosS api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-03-26T21:25:32.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: seed 79537 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (368 ms) 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (12 ms) 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (12 ms) 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (11 ms) 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-26T21:25:32.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12 ms) 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (6 ms) 2024-03-26T21:25:32.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (14 ms) 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1798 ms) 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (397 ms) 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: done waiting, doing copies 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: done waiting 2024-03-26T21:25:32.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60565 ms) 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (5 ms) 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (2 ms) 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4564 ms) 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-26T21:25:32.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: pgmap v173: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: from='client.47845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2621176687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi088-79235-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79537-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: pgmap v173: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:25:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: from='client.47845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T21:25:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2621176687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi088-79235-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79537-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: pgmap v173: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: from='client.47845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi088-79214-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2621176687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi088-79235-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79537-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2621176687' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi088-79235-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79537-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:33 smithi138 ceph-mon[33706]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T21:25:34.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2621176687' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi088-79235-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79537-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:34.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[39202]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T21:25:34.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2621176687' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi088-79235-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79537-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[30075]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T21:25:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: pgmap v176: 428 pgs: 64 unknown, 364 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1376852208' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:34 smithi138 ceph-mon[33706]: from='client.47860 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: pgmap v176: 428 pgs: 64 unknown, 364 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1376852208' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[39202]: from='client.47860 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: pgmap v176: 428 pgs: 64 unknown, 364 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T21:25:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1376852208' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:35.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:34 smithi088 ceph-mon[30075]: from='client.47860 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.47860 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2841747422' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:35 smithi138 ceph-mon[33706]: from='client.56744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.47860 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T21:25:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2841747422' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[30075]: from='client.56744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.47860 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi088-79214-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4043994525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2841747422' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]: dispatch 2024-03-26T21:25:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:35 smithi088 ceph-mon[39202]: from='client.56744 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: pgmap v179: 388 pgs: 32 creating+peering, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: from='client.56744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: pgmap v179: 388 pgs: 32 creating+peering, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: from='client.56744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: pgmap v179: 388 pgs: 32 creating+peering, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: from='client.56684 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi088-79544-16"}]': finished 2024-03-26T21:25:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: from='client.56744 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi088-79235-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T21:25:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2345666329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[30075]: from='client.47875 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T21:25:38.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:38.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2345666329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:37 smithi088 ceph-mon[39202]: from='client.47875 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T21:25:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2345666329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:38.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:37 smithi138 ceph-mon[33706]: from='client.47875 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:38 smithi138 ceph-mon[33706]: pgmap v182: 428 pgs: 72 unknown, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:38 smithi138 ceph-mon[33706]: from='client.47875 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:38 smithi138 ceph-mon[33706]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T21:25:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3262875724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:38 smithi138 ceph-mon[33706]: from='client.47881 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[30075]: pgmap v182: 428 pgs: 72 unknown, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[30075]: from='client.47875 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[30075]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3262875724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[30075]: from='client.47881 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[39202]: pgmap v182: 428 pgs: 72 unknown, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79537-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[39202]: from='client.47875 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi088-79214-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[39202]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3262875724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:38 smithi088 ceph-mon[39202]: from='client.47881 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: from='client.47881 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T21:25:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:40.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi088-79214-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:40.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:40.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: from='client.47881 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi088-79214-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: from='client.47881 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi088-79235-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:40.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi088-79214-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:40.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:25:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[39202]: pgmap v185: 460 pgs: 104 unknown, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi088-79214-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[39202]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi088-79214-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[30075]: pgmap v185: 460 pgs: 104 unknown, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:41.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi088-79214-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:41.386 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[30075]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T21:25:41.386 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi088-79214-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:41.386 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:41 smithi138 ceph-mon[33706]: pgmap v185: 460 pgs: 104 unknown, 356 active+clean; 592 KiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:41 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi088-79214-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:41 smithi138 ceph-mon[33706]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T21:25:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi088-79214-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[30075]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2390947567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[30075]: from='client.56765 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:42.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[39202]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T21:25:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2390947567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-mon[39202]: from='client.56765 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:42 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:42 smithi138 ceph-mon[33706]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T21:25:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]: dispatch 2024-03-26T21:25:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2390947567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:42 smithi138 ceph-mon[33706]: from='client.56765 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:25:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:25:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:25:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:43 smithi138 ceph-mon[33706]: pgmap v188: 364 pgs: 8 creating+peering, 356 active+clean; 592 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi088-79214-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:43 smithi138 ceph-mon[33706]: from='client.56765 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:43 smithi138 ceph-mon[33706]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T21:25:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[30075]: pgmap v188: 364 pgs: 8 creating+peering, 356 active+clean; 592 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi088-79214-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[30075]: from='client.56765 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[30075]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[39202]: pgmap v188: 364 pgs: 8 creating+peering, 356 active+clean; 592 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi088-79214-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2135315440' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79537-24"}]': finished 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[39202]: from='client.56765 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi088-79235-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:43 smithi088 ceph-mon[39202]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T21:25:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:44 smithi138 ceph-mon[33706]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T21:25:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3043369064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi088-79537-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:44 smithi088 ceph-mon[39202]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T21:25:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3043369064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi088-79537-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:44 smithi088 ceph-mon[30075]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T21:25:44.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3043369064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi088-79537-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:45 smithi138 ceph-mon[33706]: pgmap v191: 404 pgs: 40 unknown, 8 creating+peering, 356 active+clean; 592 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3043369064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi088-79537-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:45 smithi138 ceph-mon[33706]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T21:25:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1627832720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi088-79235-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[39202]: pgmap v191: 404 pgs: 40 unknown, 8 creating+peering, 356 active+clean; 592 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3043369064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi088-79537-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[39202]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1627832720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi088-79235-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[30075]: pgmap v191: 404 pgs: 40 unknown, 8 creating+peering, 356 active+clean; 592 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3043369064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi088-79537-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[30075]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:45.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1627832720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi088-79235-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1627832720' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi088-79235-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:46 smithi138 ceph-mon[33706]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T21:25:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1627832720' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi088-79235-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[39202]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1627832720' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi088-79235-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[30075]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]: dispatch 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:46.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP api_aio: Running main() from gmock_main.cc 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] Global test environment set-up. 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2955 ms) 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3002 ms) 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3023 ms) 2024-03-26T21:25:46.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3022 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2643 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3022 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3105 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3002 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2916 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2994 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3001 ms) 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-26T21:25:46.760 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2961 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3018 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3008 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.Flush (2978 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3012 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2999 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3009 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3007 ms) 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T21:25:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3008 ms) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3009 ms) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3004 ms) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3009 ms) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3010 ms) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3005 ms) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3005 ms) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 26 tests from LibRadosAio (77727 ms total) 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-26T21:25:46.762 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-26T21:25:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: pgmap v194: 428 pgs: 32 creating+peering, 396 active+clean; 620 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 4 op/s 2024-03-26T21:25:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3782116903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.56780 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:47 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: pgmap v194: 428 pgs: 32 creating+peering, 396 active+clean; 620 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 4 op/s 2024-03-26T21:25:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3782116903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.56780 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: pgmap v194: 428 pgs: 32 creating+peering, 396 active+clean; 620 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 4 op/s 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/495952316' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi088-79214-27"}]': finished 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3782116903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.56780 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:47.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:47 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.56780 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]': finished 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3831139863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:48.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:48 smithi138 ceph-mon[33706]: from='client.56786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.56780 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]': finished 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3831139863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[39202]: from='client.56786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.56780 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi088-79537-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-13"}]': finished 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi088-79214-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T21:25:48.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3831139863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:48.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:48 smithi088 ceph-mon[30075]: from='client.56786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:49 smithi138 ceph-mon[33706]: pgmap v197: 396 pgs: 32 unknown, 364 active+clean; 612 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-26T21:25:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:49 smithi138 ceph-mon[33706]: from='client.56786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:49 smithi138 ceph-mon[33706]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T21:25:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[39202]: pgmap v197: 396 pgs: 32 unknown, 364 active+clean; 612 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-26T21:25:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[39202]: from='client.56786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[39202]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T21:25:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[30075]: pgmap v197: 396 pgs: 32 unknown, 364 active+clean; 612 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-26T21:25:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[30075]: from='client.56786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi088-79235-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:49 smithi088 ceph-mon[30075]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T21:25:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-mon[33706]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T21:25:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/376346579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-mon[33706]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[30075]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T21:25:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/376346579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[30075]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi088-79214-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[39202]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/376346579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[39202]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:50 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:50.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:25:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:25:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:51 smithi138 ceph-mon[33706]: pgmap v200: 364 pgs: 32 unknown, 332 active+clean; 610 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:51 smithi138 ceph-mon[33706]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:51 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:51 smithi138 ceph-mon[33706]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T21:25:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1039556669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi088-79235-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[30075]: pgmap v200: 364 pgs: 32 unknown, 332 active+clean; 610 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[30075]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[30075]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T21:25:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1039556669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi088-79235-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[39202]: pgmap v200: 364 pgs: 32 unknown, 332 active+clean; 610 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[39202]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi088-79537-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[39202]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T21:25:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1039556669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi088-79235-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1039556669' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi088-79235-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1039556669' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi088-79235-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:52.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:52.469 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1039556669' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi088-79235-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:52.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:52 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:25:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:25:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:25:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: pgmap v203: 436 pgs: 32 unknown, 72 creating+peering, 332 active+clean; 618 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T21:25:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:53 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: pgmap v203: 436 pgs: 32 unknown, 72 creating+peering, 332 active+clean; 618 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T21:25:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: pgmap v203: 436 pgs: 32 unknown, 72 creating+peering, 332 active+clean; 618 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2962024370' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]: dispatch 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:53 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]': finished 2024-03-26T21:25:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]: dispatch 2024-03-26T21:25:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T21:25:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3798352910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi088-79235-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:54.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:54 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:54.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]': finished 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3798352910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi088-79235-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.47914 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi088-79214-28"}]': finished 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-15"}]': finished 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3798352910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi088-79235-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:54.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:54.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:54 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:25:54.779 INFO:tasks.workunit.client.0.smithi088.stdout: napshotsECPP.SnapRemovePP (2006 ms) 2024-03-26T21:25:54.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-26T21:25:54.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2009 ms) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2001 ms) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9020 ms total) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4045 ms) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3885 ms) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4021 ms) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11952 ms total) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (92569 ms total) 2024-03-26T21:25:54.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-26T21:25:54.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-26T21:25:54.781 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T21:25:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: pgmap v206: 356 pgs: 32 creating+peering, 324 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-26T21:25:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:25:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]': finished 2024-03-26T21:25:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3798352910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi088-79235-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T21:25:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:55 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: pgmap v206: 356 pgs: 32 creating+peering, 324 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-26T21:25:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]': finished 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3798352910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi088-79235-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: pgmap v206: 356 pgs: 32 creating+peering, 324 active+clean; 586 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-26T21:25:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi088-79537-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/442178140' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi088-79544-21"}]': finished 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-15", "mode": "writeback"}]': finished 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3798352910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi088-79235-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi088-79214-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:55.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:55 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:56 smithi138 ceph-mon[33706]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T21:25:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[39202]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T21:25:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[30075]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T21:25:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: pgmap v209: 396 pgs: 8 creating+peering, 388 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4088071487' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.56822 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:57.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:57 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: pgmap v209: 396 pgs: 8 creating+peering, 388 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:25:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4088071487' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.56822 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: pgmap v209: 396 pgs: 8 creating+peering, 388 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi088-79214-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4088071487' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:57.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]: dispatch 2024-03-26T21:25:57.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.56822 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:25:57.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:57 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]': finished 2024-03-26T21:25:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: from='client.56822 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:58.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:58.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T21:25:58.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:58.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:58 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]': finished 2024-03-26T21:25:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: from='client.56822 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T21:25:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-15"}]': finished 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: from='client.56822 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi088-79235-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2539369504' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:58 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]: dispatch 2024-03-26T21:25:58.764 INFO:tasks.workunit.client.0.smithi088.stdout:P (67781 ms total) 2024-03-26T21:25:58.764 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-03-26T21:25:58.764 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-26T21:25:58.764 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-26T21:25:58.764 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (115 ms) 2024-03-26T21:25:58.764 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (115 ms total) 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (26 ms) 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (11 ms) 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (37 ms total) 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (31 ms) 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (3 ms) 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (34 ms total) 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-03-26T21:25:58.765 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (45 ms) 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (5 ms) 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (50 ms total) 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1032 ms) 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1032 ms total) 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (96543 ms total) 2024-03-26T21:25:58.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: pgmap v212: 396 pgs: 40 unknown, 356 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:25:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: pgmap v212: 396 pgs: 40 unknown, 356 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:25:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T21:25:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: pgmap v212: 396 pgs: 40 unknown, 356 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi088-79537-36"}]': finished 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:25:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:25:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:00.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:26:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: pgmap v215: 324 pgs: 324 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:26:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T21:26:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:26:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1123366928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:26:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: pgmap v215: 324 pgs: 324 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:26:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T21:26:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:26:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1123366928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: pgmap v215: 324 pgs: 324 active+clean; 586 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/801483228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1123366928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]: dispatch 2024-03-26T21:26:01.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:01.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:01.789 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-26T21:26:01.789 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-26T21:26:01.789 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2960 ms) 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (18842 ms) 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5870 ms) 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3003 ms) 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3004 ms) 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2962 ms) 2024-03-26T21:26:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3013 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3003 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2985 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3014 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3005 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3006 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3006 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3007 ms) 2024-03-26T21:26:01.791 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3009 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3003 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3009 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3018 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2999 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3006 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3007 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2918 ms) 2024-03-26T21:26:01.792 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3011 ms) 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3012 ms) 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3014 ms) 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2976 ms) 2024-03-26T21:26:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-26T21:26:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:26:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T21:26:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:01 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:26:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.56816 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi088-79214-29"}]': finished 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi088-79235-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:02.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:02.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:02.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:02.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:02.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:01 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:26:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: pgmap v218: 388 pgs: 64 creating+peering, 324 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]': finished 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:02 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: pgmap v218: 388 pgs: 64 creating+peering, 324 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]': finished 2024-03-26T21:26:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T21:26:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: pgmap v218: 388 pgs: 64 creating+peering, 324 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi088-79214-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]': finished 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:03.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:02 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]': finished 2024-03-26T21:26:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]: dispatch 2024-03-26T21:26:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:03 smithi138 ceph-mon[33706]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T21:26:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]: dispatch 2024-03-26T21:26:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/405898765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:03 smithi138 ceph-mon[33706]: from='client.56840 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]': finished 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]: dispatch 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[39202]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]: dispatch 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/405898765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[39202]: from='client.56840 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:04.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-17"}]': finished 2024-03-26T21:26:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]: dispatch 2024-03-26T21:26:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[30075]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T21:26:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]: dispatch 2024-03-26T21:26:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/405898765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:03 smithi088 ceph-mon[30075]: from='client.56840 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:04.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: pgmap v221: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:04.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:04.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:04.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:04.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]': finished 2024-03-26T21:26:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: from='client.56840 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T21:26:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:04 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: pgmap v221: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]': finished 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: from='client.56840 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: pgmap v221: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi088-79214-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-17", "mode": "writeback"}]': finished 2024-03-26T21:26:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: from='client.56840 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi088-79235-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T21:26:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:04 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:05 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:05 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: pgmap v224: 364 pgs: 364 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]': finished 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2985599619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi088-79235-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2985599619' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi088-79235-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T21:26:07.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:06 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: pgmap v224: 364 pgs: 364 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T21:26:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]': finished 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2985599619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi088-79235-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2985599619' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi088-79235-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: pgmap v224: 364 pgs: 364 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T21:26:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-17"}]': finished 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2985599619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi088-79235-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2985599619' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi088-79235-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/844095747' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:07.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:06 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:08.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:07 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T21:26:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.56834 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi088-79214-30"}]': finished 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:08.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:07 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: pgmap v227: 356 pgs: 32 unknown, 324 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/427054849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.47977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:09.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: pgmap v227: 356 pgs: 32 unknown, 324 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:26:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T21:26:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/427054849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.47977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: pgmap v227: 356 pgs: 32 unknown, 324 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi088-79214-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T21:26:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:09.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/427054849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:09.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:09.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:09.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.47977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:09.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:09 smithi138 ceph-mon[33706]: from='client.47977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:09 smithi138 ceph-mon[33706]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T21:26:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[39202]: from='client.47977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[39202]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[30075]: from='client.47977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi088-79235-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[30075]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T21:26:10.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:10.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: pgmap v230: 388 pgs: 64 unknown, 324 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]': finished 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]: dispatch 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T21:26:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]: dispatch 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: pgmap v230: 388 pgs: 64 unknown, 324 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]': finished 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]: dispatch 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T21:26:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]: dispatch 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: pgmap v230: 388 pgs: 64 unknown, 324 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi088-79214-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-19"}]': finished 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]: dispatch 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T21:26:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]: dispatch 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]': finished 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T21:26:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T21:26:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3901091140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='client.56864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]': finished 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3901091140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='client.56864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-19", "mode": "writeback"}]': finished 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3901091140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='client.56864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:26:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:26:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: pgmap v233: 364 pgs: 8 unknown, 356 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T21:26:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: from='client.56864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T21:26:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:12 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: pgmap v233: 364 pgs: 8 unknown, 356 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: from='client.56864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: pgmap v233: 364 pgs: 8 unknown, 356 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: from='client.56864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi088-79235-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]: dispatch 2024-03-26T21:26:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:12 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3901091140' entity='client.admin' cmd=[{ 2024-03-26T21:26:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "prefix": "osd pool set", 2024-03-26T21:26:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "var": "eio", 2024-03-26T21:26:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "val": "true" 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: }]: dispatch 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.56864 ' entity='client.admin' cmd=[{ 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "prefix": "osd pool set", 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "var": "eio", 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "val": "true" 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: }]: dispatch 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]': finished 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.56864 ' entity='client.admin' cmd='[{ 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "prefix": "osd pool set", 2024-03-26T21:26:14.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "var": "eio", 2024-03-26T21:26:14.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: "val": "true" 2024-03-26T21:26:14.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: }]': finished 2024-03-26T21:26:14.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T21:26:14.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:14.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:13 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3901091140' entity='client.admin' cmd=[{ 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "prefix": "osd pool set", 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "var": "eio", 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "val": "true" 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: }]: dispatch 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.56864 ' entity='client.admin' cmd=[{ 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "prefix": "osd pool set", 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "var": "eio", 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "val": "true" 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: }]: dispatch 2024-03-26T21:26:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]': finished 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.56864 ' entity='client.admin' cmd='[{ 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "prefix": "osd pool set", 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "var": "eio", 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: "val": "true" 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: }]': finished 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:14.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3901091140' entity='client.admin' cmd=[{ 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "prefix": "osd pool set", 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "var": "eio", 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "val": "true" 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: }]: dispatch 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.56864 ' entity='client.admin' cmd=[{ 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "prefix": "osd pool set", 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "var": "eio", 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "val": "true" 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: }]: dispatch 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-19"}]': finished 2024-03-26T21:26:14.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.56864 ' entity='client.admin' cmd='[{ 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "prefix": "osd pool set", 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "pool": "PoolEIOFlag_smithi088-79235-33", 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "var": "eio", 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: "val": "true" 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: }]': finished 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/559017992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:14.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:13 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]: dispatch 2024-03-26T21:26:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: pgmap v236: 388 pgs: 32 unknown, 356 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T21:26:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T21:26:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:15.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:14 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: pgmap v236: 388 pgs: 32 unknown, 356 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T21:26:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: pgmap v236: 388 pgs: 32 unknown, 356 active+clean; 586 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.47983 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi088-79214-31"}]': finished 2024-03-26T21:26:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T21:26:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:14 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/650205256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.56873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:16.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:16.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:15 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T21:26:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T21:26:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/650205256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.56873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T21:26:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi088-79214-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi088-79387-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/650205256' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.56873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:16.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:15 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: pgmap v239: 323 pgs: 323 active+clean; 586 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.56873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]': finished 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:17.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T21:26:17.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:17.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:17.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: pgmap v239: 323 pgs: 323 active+clean; 586 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:26:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.56873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]': finished 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: pgmap v239: 323 pgs: 323 active+clean; 586 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:26:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.56873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi088-79235-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pg_num","val":"11"}]': finished 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:17.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T21:26:17.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T21:26:17.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:17.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: pgmap v242: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: pgmap v242: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:26:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T21:26:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T21:26:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: pgmap v242: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi088-79214-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: from='client.34712 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi088-79387-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/754332793' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]': finished 2024-03-26T21:26:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T21:26:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]: dispatch 2024-03-26T21:26:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1577496559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi088-79235-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]: dispatch 2024-03-26T21:26:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1577496559' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi088-79235-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]': finished 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]': finished 2024-03-26T21:26:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T21:26:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]: dispatch 2024-03-26T21:26:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1577496559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi088-79235-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1577496559' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi088-79235-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]': finished 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-21"}]': finished 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1577496559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi088-79235-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1577496559' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi088-79235-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-21", "mode": "writeback"}]': finished 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi088-79387-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:20.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:26:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: pgmap v245: 364 pgs: 72 unknown, 292 active+clean; 584 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: pgmap v245: 364 pgs: 72 unknown, 292 active+clean; 584 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: pgmap v245: 364 pgs: 72 unknown, 292 active+clean; 584 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:21.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:21.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/448511873' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:21.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:21.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T21:26:21.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]: dispatch 2024-03-26T21:26:21.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]: dispatch 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]': finished 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4078514771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:21 smithi138 ceph-mon[33706]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]': finished 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4078514771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[39202]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi088-79387-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: from='client.48001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi088-79214-32"}]': finished 2024-03-26T21:26:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-21"}]': finished 2024-03-26T21:26:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T21:26:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4078514771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:22.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:21 smithi088 ceph-mon[30075]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:26:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:26:22.874 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioexpected=0 2024-03-26T21:26:22.874 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:62a1935d:::14:head 2024-03-26T21:26:22.874 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:62a1935d:::14:head expected=14:62a1935d:::14:head 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:62a1935d:::14:head -> 14 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=14 expected=14 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:5c6b0b28:::7:head 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:5c6b0b28:::7:head expected=14:5c6b0b28:::7:head 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:5c6b0b28:::7:head -> 7 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=7 expected=7 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:566253c9:::13:head 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:566253c9:::13:head expected=14:566253c9:::13:head 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:566253c9:::13:head -> 13 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=13 expected=13 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:52ea6a34:::10:head 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:52ea6a34:::10:head expected=14:52ea6a34:::10:head 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:52ea6a34:::10:head -> 10 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=10 expected=10 2024-03-26T21:26:22.875 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:de5d7c5f:::12:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:de5d7c5f:::12:head expected=14:de5d7c5f:::12:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:de5d7c5f:::12:head -> 12 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=12 expected=12 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:d83876eb:::4:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:d83876eb:::4:head expected=14:d83876eb:::4:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:d83876eb:::4:head -> 4 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=4 expected=4 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:cfc208b3:::3:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:cfc208b3:::3:head expected=14:cfc208b3:::3:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:cfc208b3:::3:head -> 3 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=3 expected=3 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 14:02547ec2:::1:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=14:02547ec2:::1:head expected=14:02547ec2:::1:head 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 14:02547ec2:::1:head -> 1 2024-03-26T21:26:22.876 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=1 expected=1 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (201 ms) 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (102986 ms) 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 0/5 -> MIN 14:33333333::::head 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 1/5 -> 14:33333333::::head 14:66666666::::head 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 2/5 -> 14:66666666::::head 14:99999999::::head 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 3/5 -> 14:99999999::::head 14:cccccccc::::head 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 4/5 -> 14:cccccccc::::head MAX 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (10029 ms) 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 7 tests from LibRadosList (113831 ms total) 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-26T21:26:22.877 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1046 ms) 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo2 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo3 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo4 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo5 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo6,foo7 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo7 2024-03-26T21:26:22.878 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo6 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo4 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo5 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo7 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo6 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo1 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo1 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo2 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo3 2024-03-26T21:26:22.879 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (33 ms) 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: pgmap v248: 324 pgs: 324 active+clean; 584 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:23.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T21:26:23.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:22 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: pgmap v248: 324 pgs: 324 active+clean; 584 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:26:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: pgmap v248: 324 pgs: 324 active+clean; 584 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi088-79214-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:23.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:22 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: pgmap v251: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T21:26:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/475834179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: from='client.48031 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: pgmap v251: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T21:26:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/475834179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: from='client.48031 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: pgmap v251: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/475834179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: from='client.48031 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:25 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:25 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:25 smithi138 ceph-mon[33706]: from='client.48031 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:25 smithi138 ceph-mon[33706]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T21:26:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:25 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[39202]: from='client.48031 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[39202]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi088-79214-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[30075]: from='client.48031 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:26.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[30075]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T21:26:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3340301563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:25 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]: dispatch 2024-03-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:26 smithi138 ceph-mon[33706]: pgmap v254: 396 pgs: 8 creating+peering, 388 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 0 op/s 2024-03-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:26 smithi138 ceph-mon[33706]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:26 smithi138 ceph-mon[33706]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2893046554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:26 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:26 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[39202]: pgmap v254: 396 pgs: 8 creating+peering, 388 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 0 op/s 2024-03-26T21:26:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[39202]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[39202]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T21:26:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2893046554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[30075]: pgmap v254: 396 pgs: 8 creating+peering, 388 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 0 op/s 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[30075]: from='client.56888 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi088-79387-2"}]': finished 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[30075]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2893046554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:26 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2893046554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/557271620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.56918 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:27 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2893046554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/557271620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.56918 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2893046554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi088-79235-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T21:26:28.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/557271620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:28.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:28.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:28.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.56918 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:28.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:27 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:28.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 1 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 10 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 13 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 7 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 14 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 0 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 15 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 11 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 5 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 8 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 6 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 3 0 2024-03-26T21:26:28.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 4 0 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 12 0 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 9 0 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2 0 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (42 ms) 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 3 tests from LibRadosListEC (1121 ms total) 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3022 ms) 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 1 test from LibRadosListNP (3022 ms total) 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] Global test environment tear-down 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [==========] 11 tests from 3 test suites ran. (126664 ms total) 2024-03-26T21:26:28.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ PASSED ] 11 tests. 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: pgmap v257: 452 pgs: 64 unknown, 388 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]': finished 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.56918 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]: dispatch 2024-03-26T21:26:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]: dispatch 2024-03-26T21:26:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/557271620' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T21:26:29.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:28 smithi138 ceph-mon[33706]: from='client.56918 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T21:26:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: pgmap v257: 452 pgs: 64 unknown, 388 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:26:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]': finished 2024-03-26T21:26:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.56918 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T21:26:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]: dispatch 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]: dispatch 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/557271620' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[39202]: from='client.56918 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: pgmap v257: 452 pgs: 64 unknown, 388 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-23"}]': finished 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.56918 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79387-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T21:26:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3332054120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]: dispatch 2024-03-26T21:26:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]: dispatch 2024-03-26T21:26:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]: dispatch 2024-03-26T21:26:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/557271620' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T21:26:29.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:28 smithi088 ceph-mon[30075]: from='client.56918 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]': finished 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.56918 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:30.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:29 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]': finished 2024-03-26T21:26:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.56918 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi088-79214-33"}]': finished 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-23", "mode": "writeback"}]': finished 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.56918 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi088-79387-3","pool2":"test-rados-api-smithi088-79387-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:29 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:30.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:26:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-mon[33706]: pgmap v260: 356 pgs: 356 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-mon[33706]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T21:26:30.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:30.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:30.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:30 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[39202]: pgmap v260: 356 pgs: 356 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[39202]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T21:26:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[30075]: pgmap v260: 356 pgs: 356 active+clean; 584 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi088-79214-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[30075]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:30 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3078583587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:31 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T21:26:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3078583587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3078583587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: from='client.56930 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:31 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:26:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:26:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:32 smithi138 ceph-mon[33706]: pgmap v263: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-26T21:26:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:32 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:32 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]': finished 2024-03-26T21:26:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:32 smithi138 ceph-mon[33706]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:32 smithi138 ceph-mon[33706]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T21:26:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[39202]: pgmap v263: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-26T21:26:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]': finished 2024-03-26T21:26:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[39202]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[39202]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T21:26:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[30075]: pgmap v263: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-26T21:26:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi088-79214-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-23"}]': finished 2024-03-26T21:26:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[30075]: from='client.56930 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:32 smithi088 ceph-mon[30075]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T21:26:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:33 smithi138 ceph-mon[33706]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T21:26:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1342227764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:33 smithi088 ceph-mon[30075]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T21:26:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1342227764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:34.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:33 smithi088 ceph-mon[39202]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T21:26:34.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1342227764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:34 smithi138 ceph-mon[33706]: pgmap v266: 364 pgs: 72 unknown, 292 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1342227764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:34 smithi138 ceph-mon[33706]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T21:26:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:34 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[39202]: pgmap v266: 364 pgs: 72 unknown, 292 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1342227764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[39202]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T21:26:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[30075]: pgmap v266: 364 pgs: 72 unknown, 292 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1342227764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[30075]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:34 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3510048512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:36.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:35 smithi138 ceph-mon[33706]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T21:26:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3510048512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[30075]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T21:26:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3510048512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2172528991' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.56942 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]: dispatch 2024-03-26T21:26:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.56942 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: from='client.56924 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi088-79214-34"}]': finished 2024-03-26T21:26:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:35 smithi088 ceph-mon[39202]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: pgmap v269: 420 pgs: 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1346321859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:36 smithi138 ceph-mon[33706]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: pgmap v269: 420 pgs: 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1346321859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[39202]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: pgmap v269: 420 pgs: 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi088-79214-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1346321859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:37.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:36 smithi088 ceph-mon[30075]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:38 smithi138 ceph-mon[33706]: pgmap v272: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]': finished 2024-03-26T21:26:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:38 smithi138 ceph-mon[33706]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]: dispatch 2024-03-26T21:26:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:38 smithi138 ceph-mon[33706]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T21:26:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]: dispatch 2024-03-26T21:26:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[39202]: pgmap v272: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]': finished 2024-03-26T21:26:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[39202]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]: dispatch 2024-03-26T21:26:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[39202]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T21:26:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]: dispatch 2024-03-26T21:26:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[30075]: pgmap v272: 452 pgs: 32 unknown, 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-25"}]': finished 2024-03-26T21:26:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[30075]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]: dispatch 2024-03-26T21:26:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[30075]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T21:26:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]: dispatch 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]': finished 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2683597958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]': finished 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2683597958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi088-79214-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-25", "mode": "writeback"}]': finished 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2683597958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:40.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:40.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:26:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: pgmap v275: 492 pgs: 72 unknown, 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T21:26:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:40.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:40.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:41.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: pgmap v275: 492 pgs: 72 unknown, 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: pgmap v275: 492 pgs: 72 unknown, 32 creating+peering, 388 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi088-79235-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T21:26:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:41.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]: dispatch 2024-03-26T21:26:41.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:41.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]': finished 2024-03-26T21:26:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T21:26:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T21:26:42.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:41 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]': finished 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T21:26:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-25"}]': finished 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:42.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4116633267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T21:26:42.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:41 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]: dispatch 2024-03-26T21:26:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:26:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:26:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:42 smithi138 ceph-mon[33706]: pgmap v278: 452 pgs: 452 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:42 smithi138 ceph-mon[33706]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:42 smithi138 ceph-mon[33706]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T21:26:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[30075]: pgmap v278: 452 pgs: 452 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[30075]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[30075]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T21:26:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[39202]: pgmap v278: 452 pgs: 452 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[39202]: from='client.48067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi088-79214-35"}]': finished 2024-03-26T21:26:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[39202]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T21:26:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:43 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi088-79214-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T21:26:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:43 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:44 smithi138 ceph-mon[33706]: pgmap v281: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:44 smithi138 ceph-mon[33706]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T21:26:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:44 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[39202]: pgmap v281: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[39202]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T21:26:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[30075]: pgmap v281: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:26:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[30075]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T21:26:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:44 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: pgmap v284: 324 pgs: 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T21:26:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T21:26:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:47.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2151026273' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:47.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: from='client.56972 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:47.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:46 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: pgmap v284: 324 pgs: 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T21:26:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2151026273' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: from='client.56972 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: pgmap v284: 324 pgs: 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi088-79214-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:47.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T21:26:47.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:47.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:47.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2151026273' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:47.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: from='client.56972 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:47.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:46 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]': finished 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: from='client.56972 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]: dispatch 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]: dispatch 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]': finished 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:47 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]': finished 2024-03-26T21:26:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: from='client.56972 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]: dispatch 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]: dispatch 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]': finished 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-27"}]': finished 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: from='client.56972 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi088-79235-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T21:26:48.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]: dispatch 2024-03-26T21:26:48.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]: dispatch 2024-03-26T21:26:48.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:26:48.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-27", "mode": "writeback"}]': finished 2024-03-26T21:26:48.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T21:26:48.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:48.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:47 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:48 smithi138 ceph-mon[33706]: pgmap v287: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T21:26:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:48 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:48 smithi138 ceph-mon[33706]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T21:26:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/16778670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:48 smithi138 ceph-mon[33706]: from='client.48097 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[39202]: pgmap v287: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T21:26:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[39202]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T21:26:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/16778670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[39202]: from='client.48097 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[30075]: pgmap v287: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T21:26:49.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:49.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[30075]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T21:26:49.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/16778670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:49.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:48 smithi088 ceph-mon[30075]: from='client.48097 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.48097 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:49 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:50.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.48097 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/463993273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]: dispatch 2024-03-26T21:26:50.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:50.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:26:50.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.48097 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi088-79235-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:50.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.56966 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi088-79214-36"}]': finished 2024-03-26T21:26:50.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:26:50.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T21:26:50.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:50.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:49 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]: dispatch 2024-03-26T21:26:50.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:26:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: pgmap v290: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]': finished 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi088-79214-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:50 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: pgmap v290: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]': finished 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi088-79214-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: pgmap v290: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-27"}]': finished 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi088-79214-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:51.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:51.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:50 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:51 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:51 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:51 smithi138 ceph-mon[33706]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T21:26:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:51 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[39202]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T21:26:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:26:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[30075]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T21:26:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:51 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:26:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:26:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:52 smithi138 ceph-mon[33706]: pgmap v293: 324 pgs: 324 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:26:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi088-79214-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:52 smithi138 ceph-mon[33706]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T21:26:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:52 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[30075]: pgmap v293: 324 pgs: 324 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:26:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi088-79214-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[30075]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T21:26:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[39202]: pgmap v293: 324 pgs: 324 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:26:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi088-79214-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[39202]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T21:26:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:52 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:54 smithi138 ceph-mon[33706]: pgmap v296: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:54 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:54 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:54 smithi138 ceph-mon[33706]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T21:26:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[39202]: pgmap v296: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[39202]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T21:26:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[30075]: pgmap v296: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:26:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:54 smithi088 ceph-mon[30075]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T21:26:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:55 smithi138 ceph-mon[33706]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T21:26:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:55 smithi138 ceph-mon[33706]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T21:26:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[30075]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T21:26:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[30075]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[39202]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:55 smithi088 ceph-mon[39202]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: pgmap v299: 332 pgs: 332 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T21:26:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.127 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:26:57.127 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.127 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:26:57.127 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:57.127 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:57.127 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi088-79214-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: pgmap v299: 332 pgs: 332 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:26:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi088-79214-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: pgmap v299: 332 pgs: 332 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:26:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/438156189' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi088-79214-37"}]': finished 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/816284826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:57.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi088-79214-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: pgmap v302: 324 pgs: 324 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-03-26T21:26:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]': finished 2024-03-26T21:26:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi088-79214-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T21:26:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]: dispatch 2024-03-26T21:26:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi088-79214-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]: dispatch 2024-03-26T21:26:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:26:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:26:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:59.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:58 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: pgmap v302: 324 pgs: 324 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-03-26T21:26:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]': finished 2024-03-26T21:26:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi088-79214-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]: dispatch 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi088-79214-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]: dispatch 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: pgmap v302: 324 pgs: 324 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.56990 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-46"}]': finished 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-29"}]': finished 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi088-79214-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T21:26:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]: dispatch 2024-03-26T21:26:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi088-79214-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:26:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]: dispatch 2024-03-26T21:26:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:26:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:26:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:26:59.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:58 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:59 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:59 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]': finished 2024-03-26T21:27:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:59 smithi138 ceph-mon[33706]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T21:27:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:26:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi088-79214-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]': finished 2024-03-26T21:27:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[30075]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T21:27:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi088-79214-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-29", "mode": "writeback"}]': finished 2024-03-26T21:27:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi088-79235-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[39202]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T21:27:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:26:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi088-79214-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:00.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:27:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:27:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:00 smithi138 ceph-mon[33706]: pgmap v305: 324 pgs: 324 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:00 smithi138 ceph-mon[33706]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T21:27:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:00 smithi138 ceph-mon[33706]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T21:27:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[39202]: pgmap v305: 324 pgs: 324 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[39202]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T21:27:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[39202]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T21:27:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[30075]: pgmap v305: 324 pgs: 324 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[30075]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T21:27:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi088-79235-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:00 smithi088 ceph-mon[30075]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T21:27:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:01 smithi138 ceph-mon[33706]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T21:27:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:27:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:01 smithi088 ceph-mon[39202]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T21:27:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:27:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:01 smithi088 ceph-mon[30075]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T21:27:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:27:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:27:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: pgmap v308: 340 pgs: 8 unknown, 8 creating+peering, 324 active+clean; 8.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:27:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: pgmap v308: 340 pgs: 8 unknown, 8 creating+peering, 324 active+clean; 8.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T21:27:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: pgmap v308: 340 pgs: 8 unknown, 8 creating+peering, 324 active+clean; 8.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T21:27:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:27:03.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]: dispatch 2024-03-26T21:27:03.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]: dispatch 2024-03-26T21:27:03.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:03 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]': finished 2024-03-26T21:27:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:03 smithi138 ceph-mon[33706]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T21:27:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]': finished 2024-03-26T21:27:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[39202]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T21:27:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507050333' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi088-79214-38"}]': finished 2024-03-26T21:27:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-29"}]': finished 2024-03-26T21:27:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[30075]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T21:27:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]: dispatch 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: pgmap v311: 324 pgs: 324 active+clean; 8.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: pgmap v311: 324 pgs: 324 active+clean; 8.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T21:27:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: pgmap v311: 324 pgs: 324 active+clean; 8.6 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:05.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/683623948' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi088-79235-47"}]': finished 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi088-79214-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:05.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T21:27:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T21:27:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi088-79235-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: pgmap v314: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T21:27:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: pgmap v314: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: pgmap v314: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi088-79214-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T21:27:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:08 smithi138 ceph-mon[33706]: pgmap v317: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:08 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:08 smithi138 ceph-mon[33706]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T21:27:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[39202]: pgmap v317: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[39202]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T21:27:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[30075]: pgmap v317: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi088-79235-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[30075]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T21:27:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]': finished 2024-03-26T21:27:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:09 smithi138 ceph-mon[33706]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T21:27:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]: dispatch 2024-03-26T21:27:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]: dispatch 2024-03-26T21:27:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:09 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]': finished 2024-03-26T21:27:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[30075]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T21:27:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]: dispatch 2024-03-26T21:27:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]: dispatch 2024-03-26T21:27:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-31"}]': finished 2024-03-26T21:27:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[39202]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T21:27:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]: dispatch 2024-03-26T21:27:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]: dispatch 2024-03-26T21:27:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:09 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:27:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: pgmap v320: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]': finished 2024-03-26T21:27:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T21:27:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: pgmap v320: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]': finished 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:11.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: pgmap v320: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-31", "mode": "writeback"}]': finished 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3209624420' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]: dispatch 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:11.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi088-79214-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:11 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi088-79214-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.57008 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi088-79214-39"}]': finished 2024-03-26T21:27:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2084258245' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]: dispatch 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]: dispatch 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi088-79214-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:12.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:11 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:27:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:27:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: pgmap v323: 324 pgs: 324 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:27:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]': finished 2024-03-26T21:27:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi088-79214-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T21:27:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi088-79214-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:13.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:12 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: pgmap v323: 324 pgs: 324 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:27:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]': finished 2024-03-26T21:27:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi088-79214-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi088-79214-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: pgmap v323: 324 pgs: 324 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.57014 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi088-79235-48"}]': finished 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-31"}]': finished 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi088-79214-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi088-79214-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:13.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:12 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:13 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:13 smithi138 ceph-mon[33706]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T21:27:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:13 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:14.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi088-79214-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[30075]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T21:27:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:14.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi088-79214-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi088-79235-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[39202]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T21:27:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi088-79214-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: pgmap v326: 292 pgs: 292 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:14 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: pgmap v326: 292 pgs: 292 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T21:27:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: pgmap v326: 292 pgs: 292 active+clean; 4.6 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi088-79235-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T21:27:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:14 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:16 smithi138 ceph-mon[33706]: pgmap v329: 340 pgs: 8 creating+peering, 332 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:27:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:16 smithi138 ceph-mon[33706]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T21:27:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:16 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[30075]: pgmap v329: 340 pgs: 8 creating+peering, 332 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:27:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[30075]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T21:27:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[39202]: pgmap v329: 340 pgs: 8 creating+peering, 332 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[39202]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:16 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]': finished 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]: dispatch 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]: dispatch 2024-03-26T21:27:18.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:17 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]': finished 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]: dispatch 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]: dispatch 2024-03-26T21:27:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-33"}]': finished 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]: dispatch 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]: dispatch 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]: dispatch 2024-03-26T21:27:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:17 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: pgmap v332: 324 pgs: 324 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]': finished 2024-03-26T21:27:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T21:27:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:19.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: pgmap v332: 324 pgs: 324 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]': finished 2024-03-26T21:27:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T21:27:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: pgmap v332: 324 pgs: 324 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1516176488' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi088-79214-40"}]': finished 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-33", "mode": "writeback"}]': finished 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4058957306' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:19.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:20.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:19 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:20.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.48142 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi088-79235-49"}]': finished 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi088-79214-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:20.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:19 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:20.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:27:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-mon[33706]: pgmap v335: 324 pgs: 324 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]': finished 2024-03-26T21:27:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-mon[33706]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T21:27:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:20 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[30075]: pgmap v335: 324 pgs: 324 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]': finished 2024-03-26T21:27:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[30075]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[39202]: pgmap v335: 324 pgs: 324 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-33"}]': finished 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi088-79235-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[39202]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T21:27:21.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:20 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:21 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:21 smithi138 ceph-mon[33706]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T21:27:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:21 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[30075]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T21:27:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi088-79214-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[39202]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T21:27:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:21 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:27:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:27:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:22 smithi138 ceph-mon[33706]: pgmap v338: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:22 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:22 smithi138 ceph-mon[33706]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T21:27:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:22 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[30075]: pgmap v338: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[30075]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T21:27:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[39202]: pgmap v338: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi088-79235-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[39202]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T21:27:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:23.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:22 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:23 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:23 smithi138 ceph-mon[33706]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T21:27:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:23 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:23 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[30075]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T21:27:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:24.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[39202]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T21:27:24.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:24.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:24.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:24.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:23 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: pgmap v341: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:24 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: pgmap v341: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: pgmap v341: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3370813760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]: dispatch 2024-03-26T21:27:25.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:25.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:25.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:24 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T21:27:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]': finished 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:26.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:25 smithi138 ceph-mon[33706]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T21:27:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T21:27:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]': finished 2024-03-26T21:27:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[39202]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi088-79214-41"}]': finished 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]: dispatch 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]: dispatch 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3081010688' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]: dispatch 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-35", "mode": "writeback"}]': finished 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.57038 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi088-79235-50"}]': finished 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi088-79214-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:26.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:25 smithi088 ceph-mon[30075]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T21:27:27.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:26 smithi138 ceph-mon[33706]: pgmap v344: 324 pgs: 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:27.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:27.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:26 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:26 smithi088 ceph-mon[30075]: pgmap v344: 324 pgs: 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:26 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:26 smithi088 ceph-mon[39202]: pgmap v344: 324 pgs: 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:26 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:27 smithi138 ceph-mon[33706]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T21:27:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2944131971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:28.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:27 smithi138 ceph-mon[33706]: from='client.57050 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:27 smithi088 ceph-mon[39202]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T21:27:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2944131971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:27 smithi088 ceph-mon[39202]: from='client.57050 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:28.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:27 smithi088 ceph-mon[30075]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T21:27:28.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2944131971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:28.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:27 smithi088 ceph-mon[30075]: from='client.57050 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:28 smithi138 ceph-mon[33706]: pgmap v347: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:28 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:28 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:28 smithi138 ceph-mon[33706]: from='client.57050 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:28 smithi138 ceph-mon[33706]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T21:27:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[30075]: pgmap v347: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[30075]: from='client.57050 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[30075]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T21:27:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[39202]: pgmap v347: 356 pgs: 32 unknown, 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi088-79214-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[39202]: from='client.57050 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi088-79235-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[39202]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T21:27:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:30.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:29 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T21:27:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:30.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:30.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:29 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:30.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:27:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: pgmap v350: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T21:27:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:30.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:30 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: pgmap v350: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T21:27:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: pgmap v350: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:30 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6936 ms) 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7017 ms) 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7002 ms) 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6996 ms) 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7004 ms) 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-26T21:27:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7022 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7039 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7044 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6910 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7013 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6966 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7012 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7013 ms) 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-26T21:27:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7030 ms) 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7036 ms) 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7050 ms) 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112091 ms total) 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] Global test environment tear-down 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (189819 ms total) 2024-03-26T21:27:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:32.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T21:27:32.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:32.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2335822724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]: dispatch 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi088-79235-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.57044 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi088-79214-42"}]': finished 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:32.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]: dispatch 2024-03-26T21:27:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:27:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:27:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:32 smithi138 ceph-mon[33706]: pgmap v353: 324 pgs: 324 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:27:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:32 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:32 smithi138 ceph-mon[33706]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T21:27:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[30075]: pgmap v353: 324 pgs: 324 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:27:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[30075]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T21:27:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[39202]: pgmap v353: 324 pgs: 324 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:27:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-35"}]': finished 2024-03-26T21:27:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:32 smithi088 ceph-mon[39202]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T21:27:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:34 smithi138 ceph-mon[33706]: pgmap v356: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:27:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:34 smithi138 ceph-mon[33706]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T21:27:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:34 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:34 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[30075]: pgmap v356: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:27:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[30075]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T21:27:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[39202]: pgmap v356: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:27:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[39202]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T21:27:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:34 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2482060173' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]: dispatch 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.57056 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi088-79235-52"}]': finished 2024-03-26T21:27:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T21:27:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:36.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: pgmap v359: 324 pgs: 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T21:27:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:36 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: pgmap v359: 324 pgs: 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:27:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: pgmap v359: 324 pgs: 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:37.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:36 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:38 smithi138 ceph-mon[33706]: pgmap v362: 324 pgs: 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:27:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]': finished 2024-03-26T21:27:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]: dispatch 2024-03-26T21:27:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:38 smithi138 ceph-mon[33706]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T21:27:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]: dispatch 2024-03-26T21:27:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[39202]: pgmap v362: 324 pgs: 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:27:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]': finished 2024-03-26T21:27:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]: dispatch 2024-03-26T21:27:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[39202]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T21:27:39.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]: dispatch 2024-03-26T21:27:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[30075]: pgmap v362: 324 pgs: 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:27:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-37"}]': finished 2024-03-26T21:27:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]: dispatch 2024-03-26T21:27:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[30075]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T21:27:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]: dispatch 2024-03-26T21:27:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:39 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:39 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]': finished 2024-03-26T21:27:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:39 smithi138 ceph-mon[33706]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T21:27:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]': finished 2024-03-26T21:27:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[39202]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T21:27:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi088-79235-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-37", "mode": "writeback"}]': finished 2024-03-26T21:27:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[30075]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T21:27:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:40.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:27:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: pgmap v365: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T21:27:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:40.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:40.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:41.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: pgmap v365: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: pgmap v365: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T21:27:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:41.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]: dispatch 2024-03-26T21:27:41.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:41.136 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:41 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]': finished 2024-03-26T21:27:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:41 smithi138 ceph-mon[33706]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T21:27:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:41 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]': finished 2024-03-26T21:27:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[30075]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T21:27:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-37"}]': finished 2024-03-26T21:27:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[39202]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T21:27:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:41 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:27:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:27:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:42 smithi138 ceph-mon[33706]: pgmap v368: 324 pgs: 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T21:27:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:42 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:42 smithi138 ceph-mon[33706]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T21:27:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:42 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[30075]: pgmap v368: 324 pgs: 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T21:27:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[30075]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T21:27:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[39202]: pgmap v368: 324 pgs: 324 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T21:27:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[39202]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T21:27:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1520090507' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:43.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:42 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]: dispatch 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi088-79235-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi088-79235-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:43 smithi138 ceph-mon[33706]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T21:27:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T21:27:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi088-79235-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi088-79235-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[39202]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.57059 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi088-79235-53"}]': finished 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:44.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi088-79235-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi088-79235-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:44.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:43 smithi088 ceph-mon[30075]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: pgmap v371: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi088-79235-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T21:27:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:44 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: pgmap v371: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:27:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi088-79235-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: pgmap v371: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi088-79235-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:45.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:45.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T21:27:45.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:44 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:46 smithi138 ceph-mon[33706]: pgmap v374: 324 pgs: 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi088-79235-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]': finished 2024-03-26T21:27:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:46 smithi138 ceph-mon[33706]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T21:27:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]: dispatch 2024-03-26T21:27:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]: dispatch 2024-03-26T21:27:47.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:46 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[30075]: pgmap v374: 324 pgs: 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi088-79235-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]': finished 2024-03-26T21:27:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[30075]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T21:27:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]: dispatch 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]: dispatch 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[39202]: pgmap v374: 324 pgs: 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi088-79235-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-39"}]': finished 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[39202]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]: dispatch 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]: dispatch 2024-03-26T21:27:47.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:46 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:47 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]': finished 2024-03-26T21:27:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:47 smithi138 ceph-mon[33706]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T21:27:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]': finished 2024-03-26T21:27:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[39202]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T21:27:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:27:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-39", "mode": "writeback"}]': finished 2024-03-26T21:27:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[30075]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T21:27:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:48.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:48 smithi138 ceph-mon[33706]: pgmap v377: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:48 smithi138 ceph-mon[33706]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T21:27:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[39202]: pgmap v377: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[39202]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T21:27:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[30075]: pgmap v377: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[30075]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:49.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]: dispatch 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: seed 79477 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (392 ms) 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-26T21:27:49.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10660 ms) 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (40 ms) 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11093 ms total) 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7104 ms) 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7916 ms) 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9976 ms) 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-26T21:27:49.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [3] 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [4,3] 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting some heads 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: done waiting 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46019 ms) 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9991 ms) 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8002 ms) 2024-03-26T21:27:49.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8014 ms) 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8027 ms) 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10078 ms) 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8909 ms) 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10027 ms) 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12968 ms) 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8029 ms) 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-26T21:27:49.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8034 ms) 2024-03-26T21:27:49.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-26T21:27:49.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13067 ms) 2024-03-26T21:27:49.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-26T21:27:49.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7888 ms) 2024-03-26T21:27:49.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-26T21:27:49.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8041 ms) 2024-03-26T21:27:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:49 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:49 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]': finished 2024-03-26T21:27:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:49 smithi138 ceph-mon[33706]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T21:27:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]': finished 2024-03-26T21:27:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[39202]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T21:27:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:50.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:27:50.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-39"}]': finished 2024-03-26T21:27:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[30075]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T21:27:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]: dispatch 2024-03-26T21:27:50.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:27:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: pgmap v380: 324 pgs: 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T21:27:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:50.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:50.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:50 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: pgmap v380: 324 pgs: 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T21:27:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: pgmap v380: 324 pgs: 324 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2404872361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi088-79235-54"}]': finished 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:50 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:51 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:51 smithi138 ceph-mon[33706]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T21:27:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:51 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:51 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[39202]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T21:27:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi088-79235-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[30075]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:52.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:51 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:27:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:27:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:27:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:52 smithi138 ceph-mon[33706]: pgmap v383: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T21:27:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:52 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:52 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:52 smithi138 ceph-mon[33706]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T21:27:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:52 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[30075]: pgmap v383: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T21:27:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[30075]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T21:27:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[39202]: pgmap v383: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T21:27:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:27:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:27:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[39202]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T21:27:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:52 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:27:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:53 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:53 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:53 smithi138 ceph-mon[33706]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T21:27:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:27:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:53 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:27:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[30075]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T21:27:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:27:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:27:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi088-79235-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:54.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:27:54.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[39202]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T21:27:54.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:27:54.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:53 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:27:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:54 smithi138 ceph-mon[33706]: pgmap v386: 332 pgs: 40 unknown, 292 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T21:27:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:54 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:27:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:54 smithi138 ceph-mon[33706]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T21:27:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:27:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:54 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:27:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[39202]: pgmap v386: 332 pgs: 40 unknown, 292 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T21:27:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:27:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[39202]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T21:27:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:27:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:27:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[30075]: pgmap v386: 332 pgs: 40 unknown, 292 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T21:27:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:27:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[30075]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T21:27:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:27:55.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:54 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:27:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T21:27:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:27:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:27:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:27:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:27:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:56 smithi138 ceph-mon[33706]: pgmap v389: 324 pgs: 324 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:56 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:27:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:56 smithi138 ceph-mon[33706]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T21:27:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:56 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[39202]: pgmap v389: 324 pgs: 324 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[39202]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[30075]: pgmap v389: 324 pgs: 324 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:27:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/691586275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[30075]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T21:27:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:56 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]: dispatch 2024-03-26T21:27:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]: dispatch 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.48190 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi088-79235-55"}]': finished 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:27:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:27:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]: dispatch 2024-03-26T21:27:58.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]: dispatch 2024-03-26T21:27:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:58 smithi138 ceph-mon[33706]: pgmap v392: 324 pgs: 324 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:58 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]': finished 2024-03-26T21:27:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:58 smithi138 ceph-mon[33706]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T21:27:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:58 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[30075]: pgmap v392: 324 pgs: 324 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]': finished 2024-03-26T21:27:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[30075]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T21:27:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[39202]: pgmap v392: 324 pgs: 324 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:27:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi088-79235-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:27:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-41"}]': finished 2024-03-26T21:27:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[39202]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T21:27:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:27:59.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:58 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:27:59 smithi138 ceph-mon[33706]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T21:28:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:27:59 smithi088 ceph-mon[30075]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T21:28:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:27:59 smithi088 ceph-mon[39202]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T21:28:00.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:28:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:28:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:00 smithi138 ceph-mon[33706]: pgmap v395: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:00 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:00 smithi138 ceph-mon[33706]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T21:28:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[30075]: pgmap v395: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[30075]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T21:28:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[39202]: pgmap v395: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi088-79235-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[39202]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T21:28:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi088-79477-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T21:28:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T21:28:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T21:28:02.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:01 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T21:28:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi088-79477-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T21:28:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi088-79477-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:02.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:02.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:02.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T21:28:02.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T21:28:02.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:02.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T21:28:02.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:01 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:28:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: pgmap v398: 332 pgs: 40 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:02 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: pgmap v398: 332 pgs: 40 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T21:28:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T21:28:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: pgmap v398: 332 pgs: 40 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3403588972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:02 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]: dispatch 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: ok, hit_set contains 257:602f83fe:::foo:head 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9083 ms) 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg_num = 32 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 0 ls 1711488485,0 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 1 ls 1711488485,0 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 2 ls 1711488485,0 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 3 ls 1711488485,0 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 4 ls 1711488485,0 2024-03-26T21:28:05.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 5 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 6 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 7 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 8 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 9 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 10 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 11 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 12 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 13 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 14 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 15 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 16 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 17 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 18 ls 1711488485,0 2024-03-26T21:28:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 19 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 20 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 21 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 22 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 23 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 24 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 25 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 26 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 27 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 28 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 29 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 30 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 31 ls 1711488485,0 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg_num = 32 2024-03-26T21:28:05.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6cac518f:::0:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:02547ec2:::1:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f905c69b:::2:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:cfc208b3:::3:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d83876eb:::4:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b29083e3:::5:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c4fdafeb:::6:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:5c6b0b28:::7:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:bd63b0f1:::8:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:e960b815:::9:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:52ea6a34:::10:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:89d3ae78:::11:head 2024-03-26T21:28:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:de5d7c5f:::12:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:566253c9:::13:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:62a1935d:::14:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:863748b0:::15:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:3958e169:::16:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:4d4dabf9:::17:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:8391935d:::18:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:28883081:::19:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:69259c59:::20:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:4bdb80b7:::21:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a11c5d71:::22:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:271af37b:::23:head 2024-03-26T21:28:05.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:95b121be:::24:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:58d1031b:::25:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:0a050783:::26:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c709704c:::27:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:cbe56eaf:::28:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:86b4b162:::29:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:70d89383:::30:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:dd450c7c:::31:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6d5729b1:::32:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c388f3fb:::33:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:56cfea31:::34:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:9dbc1bf7:::35:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:40b74ccd:::36:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:4d5aaf42:::37:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:920f362c:::38:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6cc53222:::39:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:9cad833f:::40:head 2024-03-26T21:28:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:1ea84d41:::41:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c4480ef6:::42:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a694361e:::43:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d1bd33e9:::44:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:ddc2cd5d:::45:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:2b782207:::46:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:7b187fca:::47:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:90ecdf6f:::48:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a5ed95fe:::49:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:ea0eaa55:::50:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f33ef17b:::51:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a0d1b2f6:::52:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:60c5229e:::53:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:edcbc575:::54:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:102cf253:::55:head 2024-03-26T21:28:05.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:efb7fb0b:::56:head 2024-03-26T21:28:05.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:50d0a326:::57:head 2024-03-26T21:28:05.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d4dc5daf:::58:head 2024-03-26T21:28:05.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:3a130462:::59:head 2024-03-26T21:28:05.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:ec87ed71:::60:head 2024-03-26T21:28:05.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d5bc9454:::61:head 2024-03-26T21:28:05.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:3ddfe313:::62:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:7c2816b9:::63:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:47e00e4d:::64:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c6410c18:::65:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b48ed237:::66:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:cd63ad31:::67:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b179e92b:::68:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:0d9f741a:::69:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6d3352ae:::70:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c6d5c19e:::71:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:bc4729c3:::72:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:77e930b9:::73:head 2024-03-26T21:28:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:0abeecfd:::74:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b7c37e15:::75:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b6378398:::76:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:02bd68de:::77:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:cc795d2d:::78:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:630d4fea:::79:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:e0d29ef5:::80:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:fd6f13d2:::81:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:606461d5:::82:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:eadbdc43:::83:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:8761d0bb:::84:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:9ef0186f:::85:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:e0d41294:::86:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:961de695:::87:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:1423148f:::88:head 2024-03-26T21:28:05.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:633a8fa2:::89:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a8653809:::90:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:3dac8b33:::91:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:35aad435:::92:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f6dcc343:::93:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:dbbdad87:::94:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:1cb48ce0:::95:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:03cd461c:::96:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:17a4ea99:::97:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:9993c9a7:::98:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6394211c:::99:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:94c7ae57:::100:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6fdee5bb:::101:head 2024-03-26T21:28:05.170 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:9a477fd1:::102:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:eb850916:::103:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:affc56b9:::104:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b42dc814:::105:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f319f8f0:::106:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:9a40b9de:::107:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:8b524f28:::108:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:e3de589f:::109:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:90f90a5b:::110:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a7b4f1d7:::111:head 2024-03-26T21:28:05.171 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:af51766e:::112:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b6f90bd1:::113:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:e0261208:::114:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c9569ef7:::115:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:61bebe50:::116:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:fe93412b:::117:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d3d38bee:::118:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:3100ba0c:::119:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d0560ada:::120:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f0ea8b35:::121:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:766f231a:::122:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a07a2582:::123:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:bd7c6b3a:::124:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:fb2ddaff:::125:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:4408e1fe:::126:head 2024-03-26T21:28:05.172 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:ee1df7a7:::127:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c3002909:::128:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:4f48ffa9:::129:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:edf38733:::130:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c08425c0:::131:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:5f902d98:::132:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:41ea2c93:::133:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:813cee13:::134:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:0131818d:::135:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:26ba5a85:::136:head 2024-03-26T21:28:05.173 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:381b8a5a:::137:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:28797e47:::138:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:bfca7f22:::139:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:36807075:::140:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:80b03975:::141:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:5c15709b:::142:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f39ea15e:::143:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:ea992956:::144:head 2024-03-26T21:28:05.174 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:48887b1c:::145:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:9f24a9dd:::146:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:987f100b:::147:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d2dd3581:::148:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:7fed1808:::149:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c80b70e9:::150:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:85ed90f9:::151:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:36428b24:::152:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d044c34a:::153:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:7c18bf58:::154:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d1c21232:::155:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a7a3c575:::156:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:87da0633:::157:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d5ac3822:::158:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:3f20522d:::159:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6ca26563:::160:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:532ce135:::161:head 2024-03-26T21:28:05.175 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:c78863e6:::162:head 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: pgmap v401: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:05.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:04 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: pgmap v401: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:28:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: pgmap v401: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.57077 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi088-79235-56"}]': finished 2024-03-26T21:28:05.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:05.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:04 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi088-79477-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]: dispatch 2024-03-26T21:28:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]: dispatch 2024-03-26T21:28:06.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]': finished 2024-03-26T21:28:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T21:28:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T21:28:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi088-79477-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]: dispatch 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]: dispatch 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]': finished 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi088-79235-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T21:28:06.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi088-79477-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T21:28:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]: dispatch 2024-03-26T21:28:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]: dispatch 2024-03-26T21:28:06.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-43"}]': finished 2024-03-26T21:28:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:06 smithi138 ceph-mon[33706]: pgmap v404: 324 pgs: 324 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T21:28:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:06 smithi138 ceph-mon[33706]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T21:28:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:06 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:06 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:06 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:06 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:06 smithi138 ceph-mon[33706]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T21:28:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[39202]: pgmap v404: 324 pgs: 324 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T21:28:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[39202]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T21:28:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[39202]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T21:28:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[30075]: pgmap v404: 324 pgs: 324 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T21:28:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[30075]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T21:28:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi088-79235-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:06 smithi088 ceph-mon[30075]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T21:28:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:07 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:07 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:07 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:07 smithi138 ceph-mon[33706]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T21:28:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:07 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[30075]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T21:28:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[39202]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T21:28:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:07 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: pgmap v407: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: pgmap v407: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T21:28:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: pgmap v407: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:10.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:28:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: pgmap v410: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T21:28:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:28:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:28:10.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: pgmap v410: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: pgmap v410: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T21:28:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3500273810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:11.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]: dispatch 2024-03-26T21:28:11.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:28:11.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:28:11.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi088-79235-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi088-79235-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.57083 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi088-79235-57"}]': finished 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi088-79235-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:28:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: pgmap v413: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi088-79235-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi088-79235-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: pgmap v413: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi088-79235-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi088-79235-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: pgmap v413: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi088-79235-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi088-79235-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:28:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:28:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[39202]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T21:28:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:28:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi088-79235-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T21:28:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[39202]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T21:28:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:28:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[30075]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T21:28:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:28:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi088-79235-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T21:28:14.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:13 smithi088 ceph-mon[30075]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T21:28:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:28:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:13 smithi138 ceph-mon[33706]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T21:28:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:13 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:28:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi088-79235-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:13 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T21:28:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:13 smithi138 ceph-mon[33706]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T21:28:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:14 smithi088 ceph-mon[30075]: pgmap v416: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:14 smithi088 ceph-mon[30075]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T21:28:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:14 smithi088 ceph-mon[39202]: pgmap v416: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:15.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:14 smithi088 ceph-mon[39202]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T21:28:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:14 smithi138 ceph-mon[33706]: pgmap v416: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:14 smithi138 ceph-mon[33706]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T21:28:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:16 smithi138 ceph-mon[33706]: pgmap v419: 332 pgs: 332 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:28:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:16 smithi138 ceph-mon[33706]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T21:28:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:17.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:16 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[39202]: pgmap v419: 332 pgs: 332 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:28:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[39202]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T21:28:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[30075]: pgmap v419: 332 pgs: 332 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:28:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[30075]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T21:28:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:17.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:16 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[30075]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T21:28:18.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:18.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[30075]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T21:28:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[39202]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T21:28:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:17 smithi088 ceph-mon[39202]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T21:28:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:17 smithi138 ceph-mon[33706]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T21:28:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]: dispatch 2024-03-26T21:28:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1313081513' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi088-79235-58"}]': finished 2024-03-26T21:28:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:17 smithi138 ceph-mon[33706]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T21:28:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:18 smithi138 ceph-mon[33706]: pgmap v422: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:28:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:18 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:18 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:18 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[30075]: pgmap v422: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:28:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[39202]: pgmap v422: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:18 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:20 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:20 smithi138 ceph-mon[33706]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T21:28:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:20 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:20 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:20 smithi088 ceph-mon[39202]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T21:28:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:20 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:19 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:19 smithi088 ceph-mon[30075]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T21:28:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:19 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:20.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:28:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:28:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:21 smithi138 ceph-mon[33706]: pgmap v425: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:21 smithi138 ceph-mon[33706]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T21:28:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:21 smithi088 ceph-mon[39202]: pgmap v425: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:21 smithi088 ceph-mon[39202]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T21:28:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:21 smithi088 ceph-mon[30075]: pgmap v425: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:21 smithi088 ceph-mon[30075]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[39202]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[30075]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T21:28:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:22 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi088-79235-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:22 smithi138 ceph-mon[33706]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T21:28:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:22 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:28:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:28:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:28:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:23 smithi138 ceph-mon[33706]: pgmap v428: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:28:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:23 smithi138 ceph-mon[33706]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T21:28:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:23 smithi138 ceph-mon[33706]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T21:28:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:23 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[30075]: pgmap v428: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:28:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[30075]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T21:28:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[30075]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T21:28:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[39202]: pgmap v428: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:28:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[39202]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T21:28:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[39202]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T21:28:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:23 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:25 smithi138 ceph-mon[33706]: pgmap v431: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:28:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:25 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:25 smithi138 ceph-mon[33706]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T21:28:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:25 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[39202]: pgmap v431: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:28:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[39202]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T21:28:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[30075]: pgmap v431: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:28:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[30075]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T21:28:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1029387538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:25 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]: dispatch 2024-03-26T21:28:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:26 smithi138 ceph-mon[33706]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:26 smithi138 ceph-mon[33706]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T21:28:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:26 smithi088 ceph-mon[30075]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:26 smithi088 ceph-mon[30075]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T21:28:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:26 smithi088 ceph-mon[39202]: from='client.57095 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi088-79235-59"}]': finished 2024-03-26T21:28:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:26 smithi088 ceph-mon[39202]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T21:28:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:27.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: pgmap v434: 324 pgs: 324 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T21:28:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T21:28:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1832407423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: from='client.57101 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]: dispatch 2024-03-26T21:28:27.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]: dispatch 2024-03-26T21:28:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: pgmap v434: 324 pgs: 324 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T21:28:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1832407423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: from='client.57101 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: pgmap v434: 324 pgs: 324 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1832407423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: from='client.57101 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:27.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]: dispatch 2024-03-26T21:28:27.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]: dispatch 2024-03-26T21:28:28.048 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:7462ddf6:::1.RoundTripAppendPP (3026 ms) 2024-03-26T21:28:28.048 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-26T21:28:28.048 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3010 ms) 2024-03-26T21:28:28.048 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2959 ms) 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3016 ms) 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: setting pool EIO 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: max_success 99, min_failed 100 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4003 ms) 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3005 ms) 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (115682 ms total) 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-26T21:28:28.049 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3017 ms) 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9045 ms) 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14958 ms) 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2997 ms) 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30017 ms total) 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3005 ms) 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3005 ms total) 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-03-26T21:28:28.050 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13980 ms) 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7022 ms) 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7024 ms) 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7044 ms) 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3018 ms) 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7030 ms) 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6886 ms) 2024-03-26T21:28:28.051 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7037 ms) 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7053 ms) 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7033 ms) 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7027 ms) 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7015 ms) 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7048 ms) 2024-03-26T21:28:28.052 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-03-26T21:28:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:28 smithi138 ceph-mon[33706]: from='client.57101 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]': finished 2024-03-26T21:28:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:28 smithi138 ceph-mon[33706]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T21:28:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:28 smithi088 ceph-mon[30075]: from='client.57101 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]': finished 2024-03-26T21:28:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:28 smithi088 ceph-mon[30075]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T21:28:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:28 smithi088 ceph-mon[39202]: from='client.57101 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi088-79235-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-45"}]': finished 2024-03-26T21:28:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:28 smithi088 ceph-mon[39202]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: pgmap v437: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:29 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: pgmap v437: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T21:28:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T21:28:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: pgmap v437: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:29 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:30 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:30 smithi138 ceph-mon[33706]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T21:28:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:30 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:30 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[30075]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T21:28:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi088-79235-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[39202]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T21:28:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:30 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:31.059 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:28:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:28:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:31 smithi138 ceph-mon[33706]: pgmap v440: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:31 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:31 smithi138 ceph-mon[33706]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T21:28:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[30075]: pgmap v440: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[30075]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T21:28:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[39202]: pgmap v440: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[39202]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T21:28:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[39202]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T21:28:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[30075]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T21:28:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:32 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:32 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi088-79235-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:32 smithi138 ceph-mon[33706]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T21:28:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:28:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:28:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:28:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:33 smithi138 ceph-mon[33706]: pgmap v443: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:28:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:33 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]': finished 2024-03-26T21:28:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:33 smithi138 ceph-mon[33706]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T21:28:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]: dispatch 2024-03-26T21:28:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:33 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]: dispatch 2024-03-26T21:28:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[39202]: pgmap v443: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:28:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]': finished 2024-03-26T21:28:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[39202]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T21:28:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]: dispatch 2024-03-26T21:28:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]: dispatch 2024-03-26T21:28:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[30075]: pgmap v443: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:28:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-47"}]': finished 2024-03-26T21:28:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[30075]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T21:28:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]: dispatch 2024-03-26T21:28:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:33 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]: dispatch 2024-03-26T21:28:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:34 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:28:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]': finished 2024-03-26T21:28:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:34 smithi138 ceph-mon[33706]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T21:28:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:28:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:28:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:34 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:28:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]': finished 2024-03-26T21:28:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[39202]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T21:28:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:28:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:28:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:28:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-47", "mode": "writeback"}]': finished 2024-03-26T21:28:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[30075]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T21:28:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:28:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:28:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:34 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: pgmap v446: 324 pgs: 324 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: pgmap v446: 324 pgs: 324 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:28:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:28:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T21:28:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: pgmap v446: 324 pgs: 324 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2932833054' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]: dispatch 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:35.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:36.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:36 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:28:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T21:28:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.57107 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi088-79235-61"}]': finished 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:36 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: pgmap v449: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:37 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: pgmap v449: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: pgmap v449: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:37 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:28:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:28:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:38 smithi138 ceph-mon[33706]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T21:28:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:28:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:38 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T21:28:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:38 smithi138 ceph-mon[33706]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T21:28:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:28:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:28:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[39202]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T21:28:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:28:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T21:28:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[39202]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T21:28:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:28:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:28:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[30075]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T21:28:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:28:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi088-79235-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T21:28:38.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:38 smithi088 ceph-mon[30075]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T21:28:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:39 smithi138 ceph-mon[33706]: pgmap v452: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:28:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:28:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T21:28:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:39 smithi138 ceph-mon[33706]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T21:28:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[39202]: pgmap v452: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:28:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:28:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T21:28:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[39202]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T21:28:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[30075]: pgmap v452: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:28:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:28:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T21:28:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:39 smithi088 ceph-mon[30075]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:28:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T21:28:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:40 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:41.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:28:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:28:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:41 smithi138 ceph-mon[33706]: pgmap v455: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]': finished 2024-03-26T21:28:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:41 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:41 smithi138 ceph-mon[33706]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T21:28:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:41 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[30075]: pgmap v455: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]': finished 2024-03-26T21:28:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[30075]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T21:28:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[39202]: pgmap v455: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]': finished 2024-03-26T21:28:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[39202]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T21:28:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3527857006' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:41 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]: dispatch 2024-03-26T21:28:42.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[30075]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:42.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-mon[39202]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-47"}]: dispatch 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: from='client.57113 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi088-79235-62"}]': finished 2024-03-26T21:28:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:42 smithi138 ceph-mon[33706]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T21:28:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:28:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:28:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: pgmap v458: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi088-79235-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi088-79235-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi088-79235-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: pgmap v458: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:28:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi088-79235-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi088-79235-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi088-79235-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: pgmap v458: 324 pgs: 324 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi088-79235-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi088-79235-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T21:28:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi088-79235-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:28:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:45 smithi138 ceph-mon[33706]: pgmap v461: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:28:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:45 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:45 smithi138 ceph-mon[33706]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T21:28:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:45 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[30075]: pgmap v461: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:28:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[30075]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T21:28:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[39202]: pgmap v461: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:28:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:28:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[39202]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T21:28:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:45 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:28:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi088-79235-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:46 smithi138 ceph-mon[33706]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T21:28:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi088-79235-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[30075]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T21:28:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi088-79235-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:28:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[39202]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T21:28:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:47 smithi138 ceph-mon[33706]: pgmap v464: 332 pgs: 8 creating+peering, 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:28:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]': finished 2024-03-26T21:28:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:47 smithi138 ceph-mon[33706]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T21:28:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]: dispatch 2024-03-26T21:28:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]: dispatch 2024-03-26T21:28:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[30075]: pgmap v464: 332 pgs: 8 creating+peering, 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:28:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]': finished 2024-03-26T21:28:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[30075]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T21:28:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]: dispatch 2024-03-26T21:28:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]: dispatch 2024-03-26T21:28:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[39202]: pgmap v464: 332 pgs: 8 creating+peering, 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:28:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-49"}]': finished 2024-03-26T21:28:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[39202]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T21:28:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]: dispatch 2024-03-26T21:28:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]: dispatch 2024-03-26T21:28:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:48 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:28:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]': finished 2024-03-26T21:28:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:48 smithi138 ceph-mon[33706]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T21:28:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:48 smithi138 ceph-mon[33706]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]': finished 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[30075]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[30075]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-49", "mode": "readproxy"}]': finished 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[39202]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T21:28:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:48 smithi088 ceph-mon[39202]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T21:28:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:49 smithi138 ceph-mon[33706]: pgmap v467: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:28:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:49 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:49 smithi138 ceph-mon[33706]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T21:28:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[30075]: pgmap v467: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:28:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[30075]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T21:28:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[39202]: pgmap v467: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:28:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]: dispatch 2024-03-26T21:28:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2110758772' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi088-79235-63"}]': finished 2024-03-26T21:28:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:49 smithi088 ceph-mon[39202]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T21:28:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-mon[33706]: pgmap v470: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[30075]: pgmap v470: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:50 smithi088 ceph-mon[39202]: pgmap v470: 324 pgs: 324 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:51.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:28:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:28:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:51 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:51 smithi138 ceph-mon[33706]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T21:28:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:51 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[30075]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T21:28:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi088-79235-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[39202]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T21:28:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[39202]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T21:28:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[39202]: pgmap v473: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:28:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:52.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[39202]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T21:28:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[30075]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T21:28:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[30075]: pgmap v473: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:28:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-mon[30075]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T21:28:52.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:52 smithi138 ceph-mon[33706]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T21:28:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:52 smithi138 ceph-mon[33706]: pgmap v473: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:28:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:52 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi088-79235-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:52 smithi138 ceph-mon[33706]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T21:28:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:28:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:28:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:28:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:54 smithi138 ceph-mon[33706]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T21:28:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:54 smithi138 ceph-mon[33706]: pgmap v476: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:28:54.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:54 smithi088 ceph-mon[39202]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T21:28:54.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:54 smithi088 ceph-mon[39202]: pgmap v476: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:28:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:54 smithi088 ceph-mon[30075]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T21:28:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:54 smithi088 ceph-mon[30075]: pgmap v476: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:28:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:55 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:55 smithi138 ceph-mon[33706]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T21:28:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:55 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[39202]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T21:28:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[30075]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T21:28:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:55 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:56 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:56 smithi138 ceph-mon[33706]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T21:28:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:56 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:56 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:56 smithi138 ceph-mon[33706]: pgmap v479: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[39202]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T21:28:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[39202]: pgmap v479: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[30075]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T21:28:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/202951053' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]: dispatch 2024-03-26T21:28:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:28:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:28:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:56 smithi088 ceph-mon[30075]: pgmap v479: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:57 smithi138 ceph-mon[33706]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:57 smithi138 ceph-mon[33706]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T21:28:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi088-79235-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[39202]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[39202]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T21:28:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi088-79235-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[30075]: from='client.57125 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi088-79235-64"}]': finished 2024-03-26T21:28:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[30075]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T21:28:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi088-79235-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:28:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi088-79235-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:58 smithi138 ceph-mon[33706]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T21:28:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi088-79235-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:58.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:58 smithi138 ceph-mon[33706]: pgmap v482: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi088-79235-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[39202]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T21:28:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi088-79235-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[39202]: pgmap v482: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi088-79235-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:28:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[30075]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T21:28:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi088-79235-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:28:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:28:58.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:58 smithi088 ceph-mon[30075]: pgmap v482: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:28:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:59 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:28:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:59 smithi138 ceph-mon[33706]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T21:28:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:28:59 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[30075]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[39202]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T21:28:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:28:59 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout:63:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:5d165639:::164:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f43765fc:::165:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b4c720e9:::166:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:e694b040:::167:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:afa38db2:::168:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:77ba9f53:::169:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:87495034:::170:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:7c96bf0e:::171:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:dbe346cc:::172:head 2024-03-26T21:29:00.245 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:e943ec24:::173:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f97a9c0c:::174:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:6f26e74d:::175:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:4f95e106:::176:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:0e6f2f8f:::177:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:05db05f1:::178:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:38a78d66:::179:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:d095610b:::180:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:a1a9d709:::181:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:1e5d39db:::182:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:f7df4fb9:::183:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:03a7f161:::184:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:ba70721e:::185:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:28e5662d:::186:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:973d52de:::187:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:4303eb1c:::188:head 2024-03-26T21:29:00.246 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:b990b48e:::189:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:29b8165b:::190:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:3547f197:::191:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:7e260936:::192:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:1abec7b1:::193:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:10fdda93:::194:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:15817eea:::195:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:770bab57:::196:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:ed9e13e7:::197:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:71471a8f:::198:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 260:10fb1d02:::199:head 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8016 ms) 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,0 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first is 1711488494 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,0 2024-03-26T21:29:00.247 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,1711488499,1711488500,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,1711488499,1711488500,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,1711488499,1711488500,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,1711488499,1711488500,1711488503,1711488504,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,1711488499,1711488500,1711488503,1711488504,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488494,1711488496,1711488497,1711488499,1711488500,1711488503,1711488504,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711488497,1711488499,1711488500,1711488503,1711488504,1711488506,1711488507,0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first now 1711488497, trimmed 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21107 ms) 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: foo0 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14077 ms) 2024-03-26T21:29:00.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-26T21:29:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:29:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi088-79235-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]': finished 2024-03-26T21:29:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[39202]: pgmap v485: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi088-79235-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]': finished 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:00.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:29:00.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:29:00.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:00 smithi088 ceph-mon[30075]: pgmap v485: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi088-79235-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]': finished 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-49"}]: dispatch 2024-03-26T21:29:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-mon[33706]: pgmap v485: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:01.225 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:29:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:29:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:01 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:01 smithi088 ceph-mon[39202]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T21:29:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:01 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:01 smithi088 ceph-mon[30075]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T21:29:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:01.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:01 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:01 smithi138 ceph-mon[33706]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T21:29:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:02 smithi138 ceph-mon[33706]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T21:29:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:29:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:02 smithi138 ceph-mon[33706]: pgmap v488: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:29:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:02.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[39202]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T21:29:02.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:29:02.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:02.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:02.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[39202]: pgmap v488: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:02.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[30075]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T21:29:02.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:29:02.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:02.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:02.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:02 smithi088 ceph-mon[30075]: pgmap v488: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:03 smithi138 ceph-mon[33706]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T21:29:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:29:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:29:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:29:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[39202]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T21:29:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:29:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:29:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:29:03.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:03.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:03.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[30075]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T21:29:03.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]: dispatch 2024-03-26T21:29:03.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:29:03.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: pgmap v491: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]': finished 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T21:29:04.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:04.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: pgmap v491: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]': finished 2024-03-26T21:29:04.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2256091201' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi088-79235-65"}]': finished 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:04.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:29:04.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:29:04.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: pgmap v491: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:04.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-51"}]': finished 2024-03-26T21:29:04.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi088-79235-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:29:04.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T21:29:04.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]: dispatch 2024-03-26T21:29:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:05 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]: dispatch 2024-03-26T21:29:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:05 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:29:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]': finished 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]: dispatch 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]: dispatch 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]': finished 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]: dispatch 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]: dispatch 2024-03-26T21:29:05.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:29:05.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-51", "mode": "writeback"}]': finished 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: pgmap v494: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:29:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: pgmap v494: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T21:29:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: pgmap v494: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi088-79235-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:29:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:29:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:07 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:07 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:29:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:07 smithi138 ceph-mon[33706]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T21:29:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:29:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:07 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:29:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:29:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[39202]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T21:29:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:29:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:29:07.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:07.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:29:07.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[30075]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T21:29:07.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:29:07.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:07 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: pgmap v497: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:29:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T21:29:08.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:29:08.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:29:08.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:08.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:08 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: pgmap v497: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:29:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:29:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:29:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:29:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:29:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:29:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: pgmap v497: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:29:08.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:29:08.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:29:08.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T21:29:08.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:29:08.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:29:08.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:08.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:08 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3028 ms) 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7059 ms) 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7020 ms) 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7029 ms) 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-26T21:29:10.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7041 ms) 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7061 ms) 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7022 ms) 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139478 ms total) 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (288182 ms total) 2024-03-26T21:29:10.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: pgmap v500: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:10.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:29:10.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:29:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T21:29:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:29:10.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: pgmap v500: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:10.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/524787218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: pgmap v500: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-03-26T21:29:10.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T21:29:10.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:29:10.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:11.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:29:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:29:11.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:11.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-03-26T21:29:11.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]': finished 2024-03-26T21:29:11.636 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T21:29:11.636 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: from='client.57137 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi088-79235-66"}]': finished 2024-03-26T21:29:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T21:29:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-03-26T21:29:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.a", "id": [3, 1]}]': finished 2024-03-26T21:29:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-03-26T21:29:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]': finished 2024-03-26T21:29:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T21:29:11.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:12.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:12 smithi138 ceph-mon[33706]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T21:29:12.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:12 smithi138 ceph-mon[33706]: pgmap v503: 324 pgs: 4 peering, 320 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:12.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:12 smithi088 ceph-mon[30075]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T21:29:12.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:12 smithi088 ceph-mon[30075]: pgmap v503: 324 pgs: 4 peering, 320 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:12 smithi088 ceph-mon[39202]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T21:29:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:12 smithi088 ceph-mon[39202]: pgmap v503: 324 pgs: 4 peering, 320 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:12.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:29:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:29:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:13 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T21:29:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:13 smithi138 ceph-mon[33706]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T21:29:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:13 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T21:29:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:13 smithi088 ceph-mon[39202]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T21:29:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:13 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T21:29:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:13 smithi088 ceph-mon[30075]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T21:29:14.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:14 smithi138 ceph-mon[33706]: pgmap v504: 324 pgs: 4 peering, 320 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:14.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:14 smithi088 ceph-mon[39202]: pgmap v504: 324 pgs: 4 peering, 320 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:14.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:14 smithi088 ceph-mon[30075]: pgmap v504: 324 pgs: 4 peering, 320 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:15.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:15 smithi138 ceph-mon[33706]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T21:29:15.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:15 smithi088 ceph-mon[39202]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T21:29:15.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:15 smithi088 ceph-mon[30075]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T21:29:16.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:16 smithi138 ceph-mon[33706]: pgmap v505: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:16.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:16 smithi088 ceph-mon[39202]: pgmap v505: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:16.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:16 smithi088 ceph-mon[30075]: pgmap v505: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:18.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:18 smithi138 ceph-mon[33706]: pgmap v506: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:18.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:18 smithi088 ceph-mon[39202]: pgmap v506: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:18.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:18 smithi088 ceph-mon[30075]: pgmap v506: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:20.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:29:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:29:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:20 smithi138 ceph-mon[33706]: pgmap v507: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:20.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:20 smithi088 ceph-mon[39202]: pgmap v507: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:20.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:20.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:20.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:20 smithi088 ceph-mon[30075]: pgmap v507: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:20.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:20.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:21 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:29:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:21 smithi138 ceph-mon[33706]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T21:29:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:21 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:21 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]': finished 2024-03-26T21:29:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:21 smithi138 ceph-mon[33706]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T21:29:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:29:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[39202]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T21:29:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]': finished 2024-03-26T21:29:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[39202]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T21:29:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:29:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[30075]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T21:29:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]': finished 2024-03-26T21:29:21.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:21 smithi088 ceph-mon[30075]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T21:29:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[30075]: pgmap v509: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:29:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:22.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[39202]: pgmap v509: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:29:22.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:22.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:22.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:22.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:22.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:29:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:29:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:22.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:22 smithi138 ceph-mon[33706]: pgmap v509: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:29:22.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:22.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:22 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:22.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:22.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:22 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-51"}]: dispatch 2024-03-26T21:29:23.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:23 smithi138 ceph-mon[33706]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T21:29:23.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:23 smithi088 ceph-mon[30075]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T21:29:23.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:23 smithi088 ceph-mon[39202]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T21:29:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:24 smithi138 ceph-mon[33706]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T21:29:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:24 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T21:29:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:24 smithi138 ceph-mon[33706]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T21:29:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:24 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[39202]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T21:29:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T21:29:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[39202]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T21:29:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:24.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[30075]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T21:29:24.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T21:29:24.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[30075]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T21:29:24.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:24.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:24 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:25 smithi138 ceph-mon[33706]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T21:29:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-53"}]: dispatch 2024-03-26T21:29:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-53"}]: dispatch 2024-03-26T21:29:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[30075]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-53"}]: dispatch 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-53"}]: dispatch 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:25.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[39202]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T21:29:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-53"}]: dispatch 2024-03-26T21:29:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-53"}]: dispatch 2024-03-26T21:29:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:26.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:26 smithi138 ceph-mon[33706]: pgmap v515: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:29:26.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:26 smithi138 ceph-mon[33706]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T21:29:26.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:26 smithi088 ceph-mon[30075]: pgmap v515: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:29:26.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:26 smithi088 ceph-mon[30075]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T21:29:26.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:26 smithi088 ceph-mon[39202]: pgmap v515: 324 pgs: 324 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:29:26.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:26 smithi088 ceph-mon[39202]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T21:29:27.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:27 smithi138 ceph-mon[33706]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T21:29:27.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:27.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:27.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:27 smithi088 ceph-mon[39202]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T21:29:27.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:27.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:27 smithi088 ceph-mon[30075]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T21:29:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:27.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:28.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: pgmap v518: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-55"}]: dispatch 2024-03-26T21:29:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-55"}]: dispatch 2024-03-26T21:29:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: pgmap v518: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T21:29:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-55"}]: dispatch 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-55"}]: dispatch 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: pgmap v518: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-55"}]: dispatch 2024-03-26T21:29:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-55"}]: dispatch 2024-03-26T21:29:29.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:29 smithi138 ceph-mon[33706]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T21:29:29.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:29 smithi088 ceph-mon[39202]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T21:29:29.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:29 smithi088 ceph-mon[30075]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T21:29:30.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:29:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:29:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:30 smithi138 ceph-mon[33706]: pgmap v521: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:30 smithi138 ceph-mon[33706]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T21:29:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:30 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[39202]: pgmap v521: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[39202]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T21:29:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[30075]: pgmap v521: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[30075]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T21:29:30.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:30.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:30 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[39202]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T21:29:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-57"}]: dispatch 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-57"}]: dispatch 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[30075]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-57"}]: dispatch 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-57"}]: dispatch 2024-03-26T21:29:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:31 smithi138 ceph-mon[33706]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T21:29:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-57"}]: dispatch 2024-03-26T21:29:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-57"}]: dispatch 2024-03-26T21:29:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:32.725 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:29:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:29:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:32.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:32 smithi088 ceph-mon[39202]: pgmap v524: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:29:32.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:32 smithi088 ceph-mon[39202]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T21:29:32.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:32 smithi088 ceph-mon[30075]: pgmap v524: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:29:32.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:32 smithi088 ceph-mon[30075]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T21:29:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:32 smithi138 ceph-mon[33706]: pgmap v524: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:29:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:32 smithi138 ceph-mon[33706]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T21:29:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:33 smithi138 ceph-mon[33706]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T21:29:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:33 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:33 smithi088 ceph-mon[30075]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T21:29:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:33 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:33 smithi088 ceph-mon[39202]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T21:29:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:33 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: pgmap v527: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-59"}]: dispatch 2024-03-26T21:29:35.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-59"}]: dispatch 2024-03-26T21:29:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: pgmap v527: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:29:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-59"}]: dispatch 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-59"}]: dispatch 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: pgmap v527: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:35.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T21:29:35.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:35.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:35.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:35.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-59"}]: dispatch 2024-03-26T21:29:35.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-59"}]: dispatch 2024-03-26T21:29:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:35 smithi138 ceph-mon[33706]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T21:29:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:35 smithi088 ceph-mon[39202]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T21:29:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:35 smithi088 ceph-mon[30075]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T21:29:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:36 smithi138 ceph-mon[33706]: pgmap v530: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T21:29:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:36 smithi138 ceph-mon[33706]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T21:29:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[39202]: pgmap v530: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[39202]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[30075]: pgmap v530: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[30075]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:38.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:37 smithi138 ceph-mon[33706]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T21:29:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[30075]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[39202]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:38.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:38 smithi138 ceph-mon[33706]: pgmap v533: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T21:29:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:29:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:38 smithi138 ceph-mon[33706]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T21:29:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-61"}]: dispatch 2024-03-26T21:29:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-61"}]: dispatch 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[39202]: pgmap v533: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[39202]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-61"}]: dispatch 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-61"}]: dispatch 2024-03-26T21:29:39.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[30075]: pgmap v533: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T21:29:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:29:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[30075]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T21:29:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:29:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-61"}]: dispatch 2024-03-26T21:29:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-61"}]: dispatch 2024-03-26T21:29:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:39 smithi138 ceph-mon[33706]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T21:29:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:39 smithi088 ceph-mon[39202]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T21:29:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:39 smithi088 ceph-mon[30075]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T21:29:41.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[30075]: pgmap v536: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:41.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[30075]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T21:29:41.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:41.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:41.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[39202]: pgmap v536: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[39202]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T21:29:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:41.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:41.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:29:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:29:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:40 smithi138 ceph-mon[33706]: pgmap v536: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:40 smithi138 ceph-mon[33706]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T21:29:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:29:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:41 smithi138 ceph-mon[33706]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T21:29:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:41 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[39202]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T21:29:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:29:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[30075]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T21:29:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:29:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:29:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:41 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:29:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:29:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:29:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:43.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:42 smithi138 ceph-mon[33706]: pgmap v539: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:29:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:42 smithi138 ceph-mon[33706]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T21:29:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:42 smithi088 ceph-mon[39202]: pgmap v539: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:29:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:42 smithi088 ceph-mon[39202]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T21:29:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:42 smithi088 ceph-mon[30075]: pgmap v539: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:29:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:42 smithi088 ceph-mon[30075]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T21:29:44.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:43 smithi138 ceph-mon[33706]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T21:29:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:43 smithi088 ceph-mon[39202]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T21:29:44.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:43 smithi088 ceph-mon[30075]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T21:29:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:44 smithi138 ceph-mon[33706]: pgmap v542: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:44 smithi138 ceph-mon[33706]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T21:29:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:44 smithi088 ceph-mon[30075]: pgmap v542: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:44 smithi088 ceph-mon[30075]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T21:29:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:44 smithi088 ceph-mon[39202]: pgmap v542: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:29:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:44 smithi088 ceph-mon[39202]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T21:29:46.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:45 smithi138 ceph-mon[33706]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T21:29:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:45 smithi088 ceph-mon[39202]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T21:29:46.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:45 smithi088 ceph-mon[30075]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T21:29:47.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:46 smithi138 ceph-mon[33706]: pgmap v545: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-26T21:29:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:46 smithi138 ceph-mon[33706]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T21:29:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:46 smithi088 ceph-mon[30075]: pgmap v545: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-26T21:29:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:46 smithi088 ceph-mon[30075]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T21:29:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:46 smithi088 ceph-mon[39202]: pgmap v545: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-26T21:29:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:46 smithi088 ceph-mon[39202]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T21:29:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:48 smithi138 ceph-mon[33706]: pgmap v547: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-26T21:29:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:48 smithi088 ceph-mon[39202]: pgmap v547: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-26T21:29:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:48 smithi088 ceph-mon[30075]: pgmap v547: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-26T21:29:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:50 smithi138 ceph-mon[33706]: pgmap v548: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T21:29:51.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:29:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:29:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:50 smithi088 ceph-mon[30075]: pgmap v548: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T21:29:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:50 smithi088 ceph-mon[39202]: pgmap v548: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T21:29:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:29:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:29:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:53.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:52 smithi138 ceph-mon[33706]: pgmap v549: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-26T21:29:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:52 smithi088 ceph-mon[30075]: pgmap v549: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-26T21:29:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:52 smithi088 ceph-mon[39202]: pgmap v549: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-26T21:29:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:54 smithi138 ceph-mon[33706]: pgmap v550: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T21:29:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:54 smithi088 ceph-mon[39202]: pgmap v550: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T21:29:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:54 smithi088 ceph-mon[30075]: pgmap v550: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T21:29:56.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:56 smithi138 ceph-mon[33706]: pgmap v551: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:56 smithi138 ceph-mon[33706]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T21:29:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:56 smithi138 ceph-mon[33706]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T21:29:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:56 smithi088 ceph-mon[39202]: pgmap v551: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:56 smithi088 ceph-mon[39202]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T21:29:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:56 smithi088 ceph-mon[39202]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T21:29:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:56 smithi088 ceph-mon[30075]: pgmap v551: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:56 smithi088 ceph-mon[30075]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T21:29:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:56 smithi088 ceph-mon[30075]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T21:29:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:29:58 smithi138 ceph-mon[33706]: pgmap v554: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:29:58 smithi088 ceph-mon[39202]: pgmap v554: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:29:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:29:58 smithi088 ceph-mon[30075]: pgmap v554: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:00.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:30:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: pgmap v555: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:30:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T21:30:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T21:30:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: application not enabled on pool 'SnapListsmithi088-80614-1' 2024-03-26T21:30:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: application not enabled on pool 'AssertExistssmithi088-80645-1' 2024-03-26T21:30:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: application not enabled on pool 'WatchNotifysmithi088-80692-1' 2024-03-26T21:30:00.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:00 smithi138 ceph-mon[33706]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T21:30:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: pgmap v555: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:30:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T21:30:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T21:30:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: application not enabled on pool 'SnapListsmithi088-80614-1' 2024-03-26T21:30:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: application not enabled on pool 'AssertExistssmithi088-80645-1' 2024-03-26T21:30:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: application not enabled on pool 'WatchNotifysmithi088-80692-1' 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[39202]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: pgmap v555: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: application not enabled on pool 'SnapListsmithi088-80614-1' 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: application not enabled on pool 'AssertExistssmithi088-80645-1' 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: application not enabled on pool 'WatchNotifysmithi088-80692-1' 2024-03-26T21:30:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:00 smithi088 ceph-mon[30075]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T21:30:02.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:30:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:30:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:03.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:02 smithi138 ceph-mon[33706]: pgmap v556: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:02 smithi088 ceph-mon[39202]: pgmap v556: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:02 smithi088 ceph-mon[30075]: pgmap v556: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:04 smithi138 ceph-mon[33706]: pgmap v557: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:04 smithi088 ceph-mon[30075]: pgmap v557: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:04 smithi088 ceph-mon[39202]: pgmap v557: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:06 smithi138 ceph-mon[33706]: pgmap v558: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:06 smithi138 ceph-mon[33706]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T21:30:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:06 smithi138 ceph-mon[33706]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T21:30:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:06 smithi088 ceph-mon[30075]: pgmap v558: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:06 smithi088 ceph-mon[30075]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T21:30:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:06 smithi088 ceph-mon[30075]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T21:30:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:06 smithi088 ceph-mon[39202]: pgmap v558: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:06 smithi088 ceph-mon[39202]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T21:30:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:06 smithi088 ceph-mon[39202]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T21:30:09.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:08 smithi138 ceph-mon[33706]: pgmap v561: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:08 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:08 smithi088 ceph-mon[30075]: pgmap v561: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:08 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:08 smithi088 ceph-mon[39202]: pgmap v561: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:30:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:08 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:30:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:30:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:09 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:30:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:30:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:30:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:09 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:30:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:30:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:30:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:09 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:30:11.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:30:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:30:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:10 smithi138 ceph-mon[33706]: pgmap v562: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:11 smithi088 ceph-mon[39202]: pgmap v562: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:10 smithi088 ceph-mon[30075]: pgmap v562: 324 pgs: 324 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:30:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:30:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:13 smithi138 ceph-mon[33706]: pgmap v563: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T21:30:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:13 smithi088 ceph-mon[30075]: pgmap v563: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T21:30:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:13 smithi088 ceph-mon[39202]: pgmap v563: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T21:30:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:15 smithi138 ceph-mon[33706]: pgmap v564: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:30:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:15 smithi088 ceph-mon[39202]: pgmap v564: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:30:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:15 smithi088 ceph-mon[30075]: pgmap v564: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:30:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-63"}]: dispatch 2024-03-26T21:30:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-63"}]: dispatch 2024-03-26T21:30:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-63"}]: dispatch 2024-03-26T21:30:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-63"}]: dispatch 2024-03-26T21:30:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-63"}]: dispatch 2024-03-26T21:30:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-63"}]: dispatch 2024-03-26T21:30:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:17 smithi138 ceph-mon[33706]: pgmap v565: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:30:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:17 smithi138 ceph-mon[33706]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T21:30:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:17 smithi138 ceph-mon[33706]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T21:30:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:17.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:17 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[39202]: pgmap v565: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:30:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[39202]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T21:30:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[39202]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T21:30:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[30075]: pgmap v565: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:30:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[30075]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T21:30:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[30075]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T21:30:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:17.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:17 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:18 smithi138 ceph-mon[33706]: pgmap v568: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:18 smithi138 ceph-mon[33706]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T21:30:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:19.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:18 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[39202]: pgmap v568: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[39202]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T21:30:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[30075]: pgmap v568: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[30075]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T21:30:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:19.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:18 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:30:20.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:19 smithi138 ceph-mon[33706]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T21:30:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:30:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:19 smithi088 ceph-mon[39202]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T21:30:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:30:20.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:19 smithi088 ceph-mon[30075]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T21:30:21.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:30:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:30:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:20 smithi138 ceph-mon[33706]: pgmap v571: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:21.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:20 smithi138 ceph-mon[33706]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T21:30:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:20 smithi088 ceph-mon[39202]: pgmap v571: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:20 smithi088 ceph-mon[39202]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T21:30:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:20 smithi088 ceph-mon[30075]: pgmap v571: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:20 smithi088 ceph-mon[30075]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T21:30:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:21 smithi138 ceph-mon[33706]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T21:30:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:21 smithi088 ceph-mon[39202]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T21:30:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:21 smithi088 ceph-mon[30075]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T21:30:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:30:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:30:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:23.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:22 smithi138 ceph-mon[33706]: pgmap v574: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-26T21:30:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:22 smithi138 ceph-mon[33706]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T21:30:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:22 smithi088 ceph-mon[39202]: pgmap v574: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-26T21:30:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:22 smithi088 ceph-mon[39202]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T21:30:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:22 smithi088 ceph-mon[30075]: pgmap v574: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-26T21:30:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:22 smithi088 ceph-mon[30075]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T21:30:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:24 smithi138 ceph-mon[33706]: pgmap v576: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T21:30:25.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:24 smithi088 ceph-mon[30075]: pgmap v576: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T21:30:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:24 smithi088 ceph-mon[39202]: pgmap v576: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T21:30:26.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:27.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:26 smithi138 ceph-mon[33706]: pgmap v577: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-26T21:30:27.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:26 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:27.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:26 smithi138 ceph-mon[33706]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T21:30:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:26 smithi088 ceph-mon[39202]: pgmap v577: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-26T21:30:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:26 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:26 smithi088 ceph-mon[39202]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T21:30:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:26 smithi088 ceph-mon[30075]: pgmap v577: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-26T21:30:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:26 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:27.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:26 smithi088 ceph-mon[30075]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T21:30:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:28 smithi138 ceph-mon[33706]: pgmap v579: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 304 B/s wr, 3 op/s 2024-03-26T21:30:29.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:28 smithi088 ceph-mon[30075]: pgmap v579: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 304 B/s wr, 3 op/s 2024-03-26T21:30:29.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:28 smithi088 ceph-mon[39202]: pgmap v579: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 304 B/s wr, 3 op/s 2024-03-26T21:30:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:30 smithi138 ceph-mon[33706]: pgmap v580: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:30:31.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:30:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:30:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:30 smithi088 ceph-mon[30075]: pgmap v580: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:30:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:30 smithi088 ceph-mon[39202]: pgmap v580: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:30:32.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:30:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:32 smithi138 ceph-mon[33706]: pgmap v581: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 210 B/s wr, 2 op/s 2024-03-26T21:30:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-65"}]: dispatch 2024-03-26T21:30:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-65"}]: dispatch 2024-03-26T21:30:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[30075]: pgmap v581: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 210 B/s wr, 2 op/s 2024-03-26T21:30:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-65"}]: dispatch 2024-03-26T21:30:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-65"}]: dispatch 2024-03-26T21:30:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[39202]: pgmap v581: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 210 B/s wr, 2 op/s 2024-03-26T21:30:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-65"}]: dispatch 2024-03-26T21:30:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-65"}]: dispatch 2024-03-26T21:30:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:33 smithi138 ceph-mon[33706]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T21:30:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:33 smithi088 ceph-mon[39202]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T21:30:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:33 smithi088 ceph-mon[30075]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T21:30:35.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:34 smithi138 ceph-mon[33706]: pgmap v583: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:34 smithi138 ceph-mon[33706]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T21:30:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[30075]: pgmap v583: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[30075]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T21:30:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[39202]: pgmap v583: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[39202]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T21:30:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:35.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:35 smithi138 ceph-mon[33706]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T21:30:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:35 smithi088 ceph-mon[39202]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T21:30:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:35 smithi088 ceph-mon[30075]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T21:30:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:36 smithi138 ceph-mon[33706]: pgmap v586: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:36 smithi138 ceph-mon[33706]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T21:30:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-67"}]: dispatch 2024-03-26T21:30:37.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-67"}]: dispatch 2024-03-26T21:30:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[39202]: pgmap v586: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[39202]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T21:30:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-67"}]: dispatch 2024-03-26T21:30:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-67"}]: dispatch 2024-03-26T21:30:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[30075]: pgmap v586: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[30075]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T21:30:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-67"}]: dispatch 2024-03-26T21:30:37.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-67"}]: dispatch 2024-03-26T21:30:38.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:37 smithi138 ceph-mon[33706]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T21:30:38.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:37 smithi088 ceph-mon[39202]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T21:30:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:37 smithi088 ceph-mon[30075]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T21:30:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:38 smithi138 ceph-mon[33706]: pgmap v589: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:30:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:38 smithi138 ceph-mon[33706]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T21:30:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[39202]: pgmap v589: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:30:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[39202]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T21:30:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:39.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[30075]: pgmap v589: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:30:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[30075]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T21:30:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:39 smithi138 ceph-mon[33706]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T21:30:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-69"}]: dispatch 2024-03-26T21:30:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-69"}]: dispatch 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[30075]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-69"}]: dispatch 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-69"}]: dispatch 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[39202]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T21:30:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:30:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-69"}]: dispatch 2024-03-26T21:30:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-69"}]: dispatch 2024-03-26T21:30:41.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:30:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:30:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:40 smithi138 ceph-mon[33706]: pgmap v592: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:40 smithi138 ceph-mon[33706]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T21:30:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:40 smithi088 ceph-mon[39202]: pgmap v592: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:40 smithi088 ceph-mon[39202]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T21:30:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:40 smithi088 ceph-mon[30075]: pgmap v592: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:40 smithi088 ceph-mon[30075]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T21:30:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:42.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:41 smithi138 ceph-mon[33706]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T21:30:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:42.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[30075]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T21:30:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[39202]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T21:30:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:30:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:43.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:42 smithi138 ceph-mon[33706]: pgmap v595: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:30:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:42 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:42 smithi138 ceph-mon[33706]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T21:30:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[39202]: pgmap v595: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:30:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[39202]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T21:30:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[30075]: pgmap v595: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:30:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[30075]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T21:30:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:30:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:43.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:30:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:30:44.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:43 smithi138 ceph-mon[33706]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T21:30:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:30:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:43 smithi088 ceph-mon[39202]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T21:30:44.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:30:44.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:43 smithi088 ceph-mon[30075]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T21:30:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:44 smithi138 ceph-mon[33706]: pgmap v598: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:30:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:44 smithi138 ceph-mon[33706]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T21:30:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:44 smithi088 ceph-mon[30075]: pgmap v598: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:30:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:44 smithi088 ceph-mon[30075]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T21:30:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:44 smithi088 ceph-mon[39202]: pgmap v598: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:30:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:44 smithi088 ceph-mon[39202]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T21:30:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:46 smithi138 ceph-mon[33706]: pgmap v600: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-03-26T21:30:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:46 smithi088 ceph-mon[39202]: pgmap v600: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-03-26T21:30:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:46 smithi088 ceph-mon[30075]: pgmap v600: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-03-26T21:30:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:48 smithi138 ceph-mon[33706]: pgmap v601: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T21:30:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:48 smithi088 ceph-mon[39202]: pgmap v601: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T21:30:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:48 smithi088 ceph-mon[30075]: pgmap v601: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T21:30:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:50 smithi138 ceph-mon[33706]: pgmap v602: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:30:51.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:30:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:30:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:50 smithi088 ceph-mon[30075]: pgmap v602: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:30:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:50 smithi088 ceph-mon[39202]: pgmap v602: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:30:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:51 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:51 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:52.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:51 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:30:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:30:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:52 smithi088 ceph-mon[30075]: pgmap v603: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:30:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:52 smithi088 ceph-mon[39202]: pgmap v603: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:30:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:52 smithi138 ceph-mon[33706]: pgmap v603: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T21:30:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:54 smithi088 ceph-mon[30075]: pgmap v604: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T21:30:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:54 smithi088 ceph-mon[39202]: pgmap v604: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T21:30:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:54 smithi138 ceph-mon[33706]: pgmap v604: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T21:30:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:57.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:56 smithi138 ceph-mon[33706]: pgmap v605: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-26T21:30:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:56 smithi088 ceph-mon[30075]: pgmap v605: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-26T21:30:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:57 smithi088 ceph-mon[39202]: pgmap v605: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-26T21:30:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:30:59 smithi138 ceph-mon[33706]: pgmap v606: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:30:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:30:59 smithi088 ceph-mon[30075]: pgmap v606: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:30:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:30:59 smithi088 ceph-mon[39202]: pgmap v606: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:01 smithi138 ceph-mon[33706]: pgmap v607: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:01.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:31:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:31:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:01 smithi088 ceph-mon[30075]: pgmap v607: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:01 smithi088 ceph-mon[39202]: pgmap v607: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:02.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:02 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:02.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:02 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:02.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:02 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:02.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:31:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:31:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:03 smithi138 ceph-mon[33706]: pgmap v608: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:03 smithi088 ceph-mon[30075]: pgmap v608: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:03 smithi088 ceph-mon[39202]: pgmap v608: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:05.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:05 smithi138 ceph-mon[33706]: pgmap v609: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:05.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:05.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:05.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-71"}]: dispatch 2024-03-26T21:31:05.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-71"}]: dispatch 2024-03-26T21:31:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[30075]: pgmap v609: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-71"}]: dispatch 2024-03-26T21:31:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-71"}]: dispatch 2024-03-26T21:31:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[39202]: pgmap v609: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:31:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-71"}]: dispatch 2024-03-26T21:31:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-71"}]: dispatch 2024-03-26T21:31:06.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:06 smithi138 ceph-mon[33706]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T21:31:06.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:06 smithi088 ceph-mon[30075]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T21:31:06.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:06 smithi088 ceph-mon[39202]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T21:31:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:07 smithi138 ceph-mon[33706]: pgmap v611: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:07 smithi138 ceph-mon[33706]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T21:31:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:07 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[30075]: pgmap v611: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[30075]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T21:31:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[39202]: pgmap v611: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[39202]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T21:31:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:07 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:08.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:08.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:08 smithi138 ceph-mon[33706]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T21:31:08.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:08.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:08.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:08 smithi088 ceph-mon[30075]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T21:31:08.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:08.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:08.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:08 smithi088 ceph-mon[39202]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T21:31:08.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:09 smithi138 ceph-mon[33706]: pgmap v614: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:09 smithi138 ceph-mon[33706]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T21:31:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:09 smithi088 ceph-mon[39202]: pgmap v614: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:09 smithi088 ceph-mon[39202]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T21:31:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:09 smithi088 ceph-mon[30075]: pgmap v614: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:09 smithi088 ceph-mon[30075]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T21:31:10.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:10 smithi138 ceph-mon[33706]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T21:31:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:31:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:31:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:31:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:31:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[39202]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T21:31:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:31:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:31:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:31:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:31:10.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[30075]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T21:31:10.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:31:10.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:31:10.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:31:10.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:31:11.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:11 smithi138 ceph-mon[33706]: pgmap v617: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:11.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:31:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:31:11.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:11 smithi088 ceph-mon[39202]: pgmap v617: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:11.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:11 smithi088 ceph-mon[30075]: pgmap v617: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:12.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:31:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:31:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:13 smithi138 ceph-mon[33706]: pgmap v618: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T21:31:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:13 smithi088 ceph-mon[39202]: pgmap v618: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T21:31:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:13 smithi088 ceph-mon[30075]: pgmap v618: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T21:31:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:15 smithi138 ceph-mon[33706]: pgmap v619: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T21:31:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:15 smithi088 ceph-mon[39202]: pgmap v619: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T21:31:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:15 smithi088 ceph-mon[30075]: pgmap v619: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T21:31:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:17 smithi138 ceph-mon[33706]: pgmap v620: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T21:31:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:17 smithi088 ceph-mon[39202]: pgmap v620: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T21:31:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:17 smithi088 ceph-mon[30075]: pgmap v620: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T21:31:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:19 smithi138 ceph-mon[33706]: pgmap v621: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 752 B/s wr, 3 op/s 2024-03-26T21:31:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:19 smithi088 ceph-mon[39202]: pgmap v621: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 752 B/s wr, 3 op/s 2024-03-26T21:31:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:19 smithi088 ceph-mon[30075]: pgmap v621: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 752 B/s wr, 3 op/s 2024-03-26T21:31:20.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-73"}]: dispatch 2024-03-26T21:31:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-73"}]: dispatch 2024-03-26T21:31:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-73"}]: dispatch 2024-03-26T21:31:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-73"}]: dispatch 2024-03-26T21:31:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-73"}]: dispatch 2024-03-26T21:31:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-73"}]: dispatch 2024-03-26T21:31:21.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:31:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:31:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:21 smithi138 ceph-mon[33706]: pgmap v622: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 681 B/s wr, 3 op/s 2024-03-26T21:31:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:21 smithi138 ceph-mon[33706]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T21:31:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:21 smithi088 ceph-mon[39202]: pgmap v622: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 681 B/s wr, 3 op/s 2024-03-26T21:31:21.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:21 smithi088 ceph-mon[39202]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T21:31:21.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:21 smithi088 ceph-mon[30075]: pgmap v622: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 681 B/s wr, 3 op/s 2024-03-26T21:31:21.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:21 smithi088 ceph-mon[30075]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T21:31:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[30075]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T21:31:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[39202]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T21:31:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:22.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:22 smithi138 ceph-mon[33706]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T21:31:22.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:22.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:22 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:31:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:31:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:23 smithi138 ceph-mon[33706]: pgmap v625: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:31:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:23 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:23 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:23 smithi138 ceph-mon[33706]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T21:31:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[39202]: pgmap v625: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:31:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[39202]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T21:31:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[30075]: pgmap v625: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:31:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[30075]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T21:31:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:24.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:24 smithi138 ceph-mon[33706]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T21:31:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:24 smithi088 ceph-mon[30075]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T21:31:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:24 smithi088 ceph-mon[39202]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T21:31:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:25 smithi138 ceph-mon[33706]: pgmap v628: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:25 smithi138 ceph-mon[33706]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T21:31:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-75"}]: dispatch 2024-03-26T21:31:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-75"}]: dispatch 2024-03-26T21:31:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[39202]: pgmap v628: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[39202]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T21:31:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-75"}]: dispatch 2024-03-26T21:31:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-75"}]: dispatch 2024-03-26T21:31:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[30075]: pgmap v628: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[30075]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T21:31:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-75"}]: dispatch 2024-03-26T21:31:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-75"}]: dispatch 2024-03-26T21:31:26.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:26 smithi138 ceph-mon[33706]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T21:31:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:26 smithi088 ceph-mon[39202]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T21:31:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:26 smithi088 ceph-mon[30075]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T21:31:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:27 smithi138 ceph-mon[33706]: pgmap v631: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:31:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:27 smithi138 ceph-mon[33706]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T21:31:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[39202]: pgmap v631: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:31:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[39202]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T21:31:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[30075]: pgmap v631: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:31:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[30075]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T21:31:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:28 smithi138 ceph-mon[33706]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T21:31:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:28 smithi088 ceph-mon[30075]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T21:31:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:28 smithi088 ceph-mon[39202]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T21:31:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:29 smithi138 ceph-mon[33706]: pgmap v634: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:31:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:29 smithi138 ceph-mon[33706]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T21:31:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-77"}]: dispatch 2024-03-26T21:31:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-77"}]: dispatch 2024-03-26T21:31:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[30075]: pgmap v634: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:31:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[30075]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T21:31:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-77"}]: dispatch 2024-03-26T21:31:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-77"}]: dispatch 2024-03-26T21:31:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[39202]: pgmap v634: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:31:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[39202]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T21:31:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-77"}]: dispatch 2024-03-26T21:31:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-77"}]: dispatch 2024-03-26T21:31:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:30 smithi138 ceph-mon[33706]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T21:31:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:30 smithi088 ceph-mon[30075]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T21:31:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:30 smithi088 ceph-mon[39202]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T21:31:31.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:31:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:31:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:31 smithi138 ceph-mon[33706]: pgmap v637: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:31 smithi138 ceph-mon[33706]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T21:31:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[30075]: pgmap v637: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[30075]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T21:31:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[39202]: pgmap v637: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:31:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[39202]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T21:31:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:32.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:32.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:32.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[39202]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T21:31:32.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:32.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:32.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[30075]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T21:31:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:32.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:32 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:32.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:32.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:32 smithi138 ceph-mon[33706]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T21:31:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:32 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:31:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:31:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:33 smithi138 ceph-mon[33706]: pgmap v640: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:31:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:33 smithi138 ceph-mon[33706]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T21:31:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:33 smithi088 ceph-mon[39202]: pgmap v640: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:31:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:33 smithi088 ceph-mon[39202]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T21:31:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:33 smithi088 ceph-mon[30075]: pgmap v640: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:31:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:33 smithi088 ceph-mon[30075]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T21:31:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:34 smithi138 ceph-mon[33706]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T21:31:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "288.0"}]: dispatch 2024-03-26T21:31:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:34 smithi138 ceph-mon[33706]: 288.0 deep-scrub starts 2024-03-26T21:31:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:34 smithi138 ceph-mon[33706]: 288.0 deep-scrub ok 2024-03-26T21:31:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[39202]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T21:31:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "288.0"}]: dispatch 2024-03-26T21:31:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[39202]: 288.0 deep-scrub starts 2024-03-26T21:31:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[39202]: 288.0 deep-scrub ok 2024-03-26T21:31:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[30075]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T21:31:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "288.0"}]: dispatch 2024-03-26T21:31:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[30075]: 288.0 deep-scrub starts 2024-03-26T21:31:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:34 smithi088 ceph-mon[30075]: 288.0 deep-scrub ok 2024-03-26T21:31:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:35 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "288.0"}]: dispatch 2024-03-26T21:31:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:35 smithi138 ceph-mon[33706]: pgmap v643: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:31:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:35 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "288.0"}]: dispatch 2024-03-26T21:31:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:35 smithi088 ceph-mon[39202]: pgmap v643: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:31:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:35 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "288.0"}]: dispatch 2024-03-26T21:31:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:35 smithi088 ceph-mon[30075]: pgmap v643: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:31:36.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:36 smithi138 ceph-mon[33706]: pgmap v644: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 1.3 KiB/s wr, 136 op/s 2024-03-26T21:31:36.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:36 smithi088 ceph-mon[30075]: pgmap v644: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 1.3 KiB/s wr, 136 op/s 2024-03-26T21:31:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:36 smithi088 ceph-mon[39202]: pgmap v644: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 1.3 KiB/s wr, 136 op/s 2024-03-26T21:31:38.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:38 smithi138 ceph-mon[33706]: pgmap v645: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 798 B/s wr, 113 op/s 2024-03-26T21:31:38.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:38 smithi088 ceph-mon[39202]: pgmap v645: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 798 B/s wr, 113 op/s 2024-03-26T21:31:38.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:38 smithi088 ceph-mon[30075]: pgmap v645: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 798 B/s wr, 113 op/s 2024-03-26T21:31:40.904 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:40 smithi138 ceph-mon[33706]: pgmap v646: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 639 B/s wr, 91 op/s 2024-03-26T21:31:40.904 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:40.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:40 smithi088 ceph-mon[30075]: pgmap v646: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 639 B/s wr, 91 op/s 2024-03-26T21:31:40.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:40.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:40 smithi088 ceph-mon[39202]: pgmap v646: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 639 B/s wr, 91 op/s 2024-03-26T21:31:40.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:41.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:31:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:31:41.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:41.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:42 smithi088 ceph-mon[30075]: pgmap v647: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 544 B/s wr, 78 op/s 2024-03-26T21:31:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:42 smithi088 ceph-mon[39202]: pgmap v647: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 544 B/s wr, 78 op/s 2024-03-26T21:31:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:31:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:31:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:42.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:42 smithi138 ceph-mon[33706]: pgmap v647: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 544 B/s wr, 78 op/s 2024-03-26T21:31:44.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:44 smithi138 ceph-mon[33706]: pgmap v648: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 492 B/s wr, 70 op/s 2024-03-26T21:31:44.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:44 smithi088 ceph-mon[30075]: pgmap v648: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 492 B/s wr, 70 op/s 2024-03-26T21:31:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:44 smithi088 ceph-mon[39202]: pgmap v648: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 492 B/s wr, 70 op/s 2024-03-26T21:31:46.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:46 smithi138 ceph-mon[33706]: pgmap v649: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-26T21:31:46.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:46 smithi088 ceph-mon[30075]: pgmap v649: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-26T21:31:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:46 smithi088 ceph-mon[39202]: pgmap v649: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-26T21:31:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:48 smithi138 ceph-mon[33706]: pgmap v650: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:48 smithi088 ceph-mon[30075]: pgmap v650: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:48 smithi088 ceph-mon[39202]: pgmap v650: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:50 smithi138 ceph-mon[33706]: pgmap v651: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:51.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:31:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:31:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:50 smithi088 ceph-mon[30075]: pgmap v651: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:50 smithi088 ceph-mon[39202]: pgmap v651: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:52.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:31:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:31:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:53.216 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:52 smithi088 ceph-mon[39202]: pgmap v652: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:31:53.216 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:52 smithi088 ceph-mon[30075]: pgmap v652: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:31:53.217 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:52 smithi138 ceph-mon[33706]: pgmap v652: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:31:53.474 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:31:53 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:31:53.223+0000 7f4250043700 -1 snap_mapper.add_oid found existing snaps mapped on 288:0584d68f:test-rados-api-smithi088-79477-80::foo:2, removing 2024-03-26T21:31:53.483 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0[43605]: 2024-03-26T21:31:53.223+0000 7f0ede21d700 -1 snap_mapper.add_oid found existing snaps mapped on 288:0584d68f:test-rados-api-smithi088-79477-80::foo:2, removing 2024-03-26T21:31:53.483 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1[49547]: 2024-03-26T21:31:53.223+0000 7f42a8512700 -1 snap_mapper.add_oid found existing snaps mapped on 288:0584d68f:test-rados-api-smithi088-79477-80::foo:2, removing 2024-03-26T21:31:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:53 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-79"}]: dispatch 2024-03-26T21:31:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:53 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-79"}]: dispatch 2024-03-26T21:31:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-79"}]: dispatch 2024-03-26T21:31:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-79"}]: dispatch 2024-03-26T21:31:54.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:54.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:54.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-79"}]: dispatch 2024-03-26T21:31:54.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:53 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-79"}]: dispatch 2024-03-26T21:31:55.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:54 smithi138 ceph-mon[33706]: pgmap v653: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:55.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:54 smithi138 ceph-mon[33706]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T21:31:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:54 smithi088 ceph-mon[30075]: pgmap v653: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:54 smithi088 ceph-mon[30075]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T21:31:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:54 smithi088 ceph-mon[39202]: pgmap v653: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:31:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:54 smithi088 ceph-mon[39202]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T21:31:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:55 smithi138 ceph-mon[33706]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T21:31:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:55 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[30075]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T21:31:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[39202]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T21:31:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:31:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:56 smithi138 ceph-mon[33706]: pgmap v656: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T21:31:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:56 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:56 smithi138 ceph-mon[33706]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T21:31:57.125 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:31:57.126 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:56 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:31:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[39202]: pgmap v656: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T21:31:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[39202]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T21:31:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[30075]: pgmap v656: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[30075]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:31:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:56 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:31:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:31:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:57 smithi138 ceph-mon[33706]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T21:31:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:57 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:31:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[39202]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T21:31:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:31:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[30075]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T21:31:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:57 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:31:59.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:58 smithi138 ceph-mon[33706]: pgmap v659: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T21:31:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:31:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:58 smithi138 ceph-mon[33706]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T21:31:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:31:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:58 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:31:59.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[39202]: pgmap v659: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T21:31:59.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:31:59.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[39202]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T21:31:59.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:31:59.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:31:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[30075]: pgmap v659: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T21:31:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:31:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[30075]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T21:31:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:31:59.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:58 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:00.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:59 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:59 smithi138 ceph-mon[33706]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T21:32:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:31:59 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[30075]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T21:32:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[39202]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T21:32:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:31:59 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:01.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:32:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:32:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:00 smithi138 ceph-mon[33706]: pgmap v662: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:00 smithi138 ceph-mon[33706]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T21:32:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T21:32:01.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:00 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T21:32:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[39202]: pgmap v662: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[39202]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T21:32:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T21:32:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T21:32:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[30075]: pgmap v662: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[30075]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T21:32:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T21:32:01.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:00 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T21:32:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T21:32:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:01 smithi138 ceph-mon[33706]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T21:32:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T21:32:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:01 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T21:32:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:01 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T21:32:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[39202]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T21:32:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T21:32:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T21:32:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T21:32:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[30075]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T21:32:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T21:32:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T21:32:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:02.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:01 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:32:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:03.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:02 smithi138 ceph-mon[33706]: pgmap v665: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-26T21:32:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T21:32:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:02 smithi138 ceph-mon[33706]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T21:32:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:02 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[39202]: pgmap v665: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-26T21:32:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T21:32:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[39202]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T21:32:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:03.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[30075]: pgmap v665: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-26T21:32:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T21:32:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[30075]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T21:32:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:03.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:02 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:04.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:03 smithi138 ceph-mon[33706]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T21:32:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-81"}]: dispatch 2024-03-26T21:32:04.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:03 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-81"}]: dispatch 2024-03-26T21:32:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[39202]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T21:32:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-81"}]: dispatch 2024-03-26T21:32:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-81"}]: dispatch 2024-03-26T21:32:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[30075]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T21:32:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-81"}]: dispatch 2024-03-26T21:32:04.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:03 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-81"}]: dispatch 2024-03-26T21:32:05.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:04 smithi138 ceph-mon[33706]: pgmap v668: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-26T21:32:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:04 smithi138 ceph-mon[33706]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T21:32:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:04 smithi088 ceph-mon[39202]: pgmap v668: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-26T21:32:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:04 smithi088 ceph-mon[39202]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T21:32:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:04 smithi088 ceph-mon[30075]: pgmap v668: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-26T21:32:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:04 smithi088 ceph-mon[30075]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T21:32:06.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:05 smithi138 ceph-mon[33706]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T21:32:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:06.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:05 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:05 smithi088 ceph-mon[39202]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T21:32:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:05 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:05 smithi088 ceph-mon[30075]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T21:32:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:06.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:05 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:07.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:06 smithi138 ceph-mon[33706]: pgmap v671: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-26T21:32:07.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:06 smithi138 ceph-mon[33706]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T21:32:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:06 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[39202]: pgmap v671: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-26T21:32:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[39202]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T21:32:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[30075]: pgmap v671: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[30075]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:06 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:08.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:07 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:07 smithi138 ceph-mon[33706]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T21:32:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:08.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:07 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[30075]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T21:32:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[39202]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T21:32:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:07 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:08 smithi138 ceph-mon[33706]: pgmap v674: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-26T21:32:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:32:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:08 smithi138 ceph-mon[33706]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T21:32:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[30075]: pgmap v674: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-26T21:32:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:32:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[30075]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T21:32:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[39202]: pgmap v674: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-26T21:32:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:32:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[39202]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T21:32:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:10.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:10.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:09 smithi138 ceph-mon[33706]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T21:32:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:09 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:32:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:09 smithi138 ceph-mon[33706]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[30075]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[30075]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T21:32:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[39202]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T21:32:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:32:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:09 smithi088 ceph-mon[39202]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T21:32:11.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:32:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:32:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:10 smithi138 ceph-mon[33706]: pgmap v677: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:32:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:32:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:32:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[39202]: pgmap v677: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:32:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:32:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:32:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[30075]: pgmap v677: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:32:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:32:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:32:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:11 smithi088 ceph-mon[39202]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T21:32:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:11 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:11 smithi088 ceph-mon[30075]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T21:32:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:11 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:11 smithi138 ceph-mon[33706]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T21:32:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:11 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:32:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:12 smithi138 ceph-mon[33706]: pgmap v680: 324 pgs: 324 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T21:32:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:12 smithi138 ceph-mon[33706]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T21:32:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-83"}]: dispatch 2024-03-26T21:32:13.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:12 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-83"}]: dispatch 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[30075]: pgmap v680: 324 pgs: 324 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[30075]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-83"}]: dispatch 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-83"}]: dispatch 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[39202]: pgmap v680: 324 pgs: 324 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T21:32:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[39202]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T21:32:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-83"}]: dispatch 2024-03-26T21:32:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:12 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-83"}]: dispatch 2024-03-26T21:32:14.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:13 smithi138 ceph-mon[33706]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T21:32:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:13 smithi088 ceph-mon[39202]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T21:32:14.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:13 smithi088 ceph-mon[30075]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T21:32:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:15 smithi138 ceph-mon[33706]: pgmap v683: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:15 smithi138 ceph-mon[33706]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T21:32:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:15 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[30075]: pgmap v683: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[30075]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T21:32:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:15.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[39202]: pgmap v683: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[39202]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T21:32:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:15 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:16.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:16 smithi138 ceph-mon[33706]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T21:32:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:16 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[30075]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T21:32:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[39202]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T21:32:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:16 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:17 smithi138 ceph-mon[33706]: pgmap v686: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-26T21:32:17.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:17 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:17.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:17 smithi138 ceph-mon[33706]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T21:32:17.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:17.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:17 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[39202]: pgmap v686: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-26T21:32:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[39202]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T21:32:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[30075]: pgmap v686: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-26T21:32:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[30075]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T21:32:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:17 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:32:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:18 smithi138 ceph-mon[33706]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T21:32:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:18 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:32:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[39202]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T21:32:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_tier","val": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:32:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[30075]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T21:32:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:18 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:19 smithi138 ceph-mon[33706]: pgmap v689: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T21:32:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:19 smithi138 ceph-mon[33706]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T21:32:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:19 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[39202]: pgmap v689: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T21:32:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[39202]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T21:32:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[30075]: pgmap v689: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T21:32:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[30075]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T21:32:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:19 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:20.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:20 smithi138 ceph-mon[33706]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T21:32:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:20 smithi088 ceph-mon[30075]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T21:32:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:20 smithi088 ceph-mon[39202]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T21:32:21.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:32:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:32:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:21 smithi138 ceph-mon[33706]: pgmap v692: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:21 smithi138 ceph-mon[33706]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T21:32:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:21 smithi088 ceph-mon[30075]: pgmap v692: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:21 smithi088 ceph-mon[30075]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T21:32:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:21 smithi088 ceph-mon[39202]: pgmap v692: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:21 smithi088 ceph-mon[39202]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T21:32:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[30075]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T21:32:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-85"}]: dispatch 2024-03-26T21:32:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-85"}]: dispatch 2024-03-26T21:32:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[39202]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T21:32:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-85"}]: dispatch 2024-03-26T21:32:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-85"}]: dispatch 2024-03-26T21:32:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:22 smithi138 ceph-mon[33706]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T21:32:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:22 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-85"}]: dispatch 2024-03-26T21:32:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:22 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-85"}]: dispatch 2024-03-26T21:32:22.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:32:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:32:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:23 smithi138 ceph-mon[33706]: pgmap v695: 324 pgs: 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-26T21:32:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:23 smithi138 ceph-mon[33706]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T21:32:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:23 smithi088 ceph-mon[30075]: pgmap v695: 324 pgs: 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-26T21:32:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:23 smithi088 ceph-mon[30075]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T21:32:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:23 smithi088 ceph-mon[39202]: pgmap v695: 324 pgs: 324 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-26T21:32:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:23 smithi088 ceph-mon[39202]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T21:32:24.094 INFO:tasks.workunit.client.0.smithi088.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18122 ms) 2024-03-26T21:32:24.094 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-26T21:32:24.094 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22417 ms) 2024-03-26T21:32:24.094 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-26T21:32:24.094 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3022 ms) 2024-03-26T21:32:24.094 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3021 ms) 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3025 ms) 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3022 ms) 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4024 ms) 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (37243 ms) 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-26T21:32:24.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16853 ms) 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4037 ms) 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3026 ms) 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25080 ms) 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15072 ms) 2024-03-26T21:32:24.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5030 ms) 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4037 ms) 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: done waiting 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24691 ms) 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10078 ms) 2024-03-26T21:32:24.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-26T21:32:24.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9061 ms) 2024-03-26T21:32:24.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-26T21:32:24.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:32:24.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9094 ms) 2024-03-26T21:32:24.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-26T21:32:24.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:24 smithi138 ceph-mon[33706]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T21:32:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:24 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:24 smithi088 ceph-mon[39202]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T21:32:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:24 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:24.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:24 smithi088 ceph-mon[30075]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T21:32:24.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:24.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:24 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:25 smithi138 ceph-mon[33706]: pgmap v698: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:32:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:25 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:25 smithi138 ceph-mon[33706]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T21:32:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[39202]: pgmap v698: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:32:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[39202]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T21:32:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[30075]: pgmap v698: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:32:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[30075]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T21:32:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:26.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:26 smithi138 ceph-mon[33706]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T21:32:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:26 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[39202]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T21:32:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[30075]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T21:32:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:27.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:27 smithi138 ceph-mon[33706]: pgmap v701: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-03-26T21:32:27.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:27.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:27 smithi138 ceph-mon[33706]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T21:32:27.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]: dispatch 2024-03-26T21:32:27.375 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]: dispatch 2024-03-26T21:32:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[39202]: pgmap v701: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-03-26T21:32:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[39202]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T21:32:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]: dispatch 2024-03-26T21:32:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]: dispatch 2024-03-26T21:32:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[30075]: pgmap v701: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-03-26T21:32:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[30075]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T21:32:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]: dispatch 2024-03-26T21:32:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]: dispatch 2024-03-26T21:32:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]': finished 2024-03-26T21:32:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:28 smithi138 ceph-mon[33706]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T21:32:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]': finished 2024-03-26T21:32:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[39202]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T21:32:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_tier","val": "test-rados-api-smithi088-79477-89-test-flush"}]': finished 2024-03-26T21:32:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[30075]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T21:32:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:28.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:32:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:29 smithi138 ceph-mon[33706]: pgmap v704: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-26T21:32:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:29 smithi138 ceph-mon[33706]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T21:32:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[39202]: pgmap v704: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-26T21:32:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[39202]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[30075]: pgmap v704: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[30075]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:32:30.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:30 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:30 smithi138 ceph-mon[33706]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T21:32:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:30 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:30 smithi088 ceph-mon[39202]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T21:32:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:30 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:32:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:30 smithi088 ceph-mon[30075]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T21:32:31.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:32:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:32:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:31 smithi138 ceph-mon[33706]: pgmap v707: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:31 smithi138 ceph-mon[33706]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T21:32:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-87"}]: dispatch 2024-03-26T21:32:31.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-87"}]: dispatch 2024-03-26T21:32:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[30075]: pgmap v707: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[30075]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T21:32:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-87"}]: dispatch 2024-03-26T21:32:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-87"}]: dispatch 2024-03-26T21:32:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[39202]: pgmap v707: 356 pgs: 32 creating+peering, 324 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[39202]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T21:32:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-87"}]: dispatch 2024-03-26T21:32:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-87"}]: dispatch 2024-03-26T21:32:32.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:32 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:32.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:32 smithi088 ceph-mon[30075]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T21:32:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:32 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:32 smithi088 ceph-mon[39202]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T21:32:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:32 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:32 smithi138 ceph-mon[33706]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T21:32:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:32:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:32:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:33 smithi138 ceph-mon[33706]: pgmap v710: 292 pgs: 292 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:33 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:33 smithi138 ceph-mon[33706]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T21:32:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:33 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[39202]: pgmap v710: 292 pgs: 292 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[39202]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T21:32:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[30075]: pgmap v710: 292 pgs: 292 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[30075]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T21:32:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:33.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:33 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:34.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:34 smithi138 ceph-mon[33706]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T21:32:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:34.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:34.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[30075]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[39202]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:32:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:35 smithi138 ceph-mon[33706]: pgmap v713: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:35.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:35 smithi138 ceph-mon[33706]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T21:32:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:35 smithi088 ceph-mon[30075]: pgmap v713: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:35 smithi088 ceph-mon[30075]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T21:32:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:35 smithi088 ceph-mon[39202]: pgmap v713: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:32:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:32:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:35 smithi088 ceph-mon[39202]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T21:32:36.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:36 smithi138 ceph-mon[33706]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T21:32:36.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:36 smithi088 ceph-mon[30075]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T21:32:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:36 smithi088 ceph-mon[39202]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T21:32:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:37 smithi138 ceph-mon[33706]: pgmap v716: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-26T21:32:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:37 smithi138 ceph-mon[33706]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T21:32:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-90"}]: dispatch 2024-03-26T21:32:37.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-90"}]: dispatch 2024-03-26T21:32:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[30075]: pgmap v716: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-26T21:32:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[30075]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T21:32:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-90"}]: dispatch 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-90"}]: dispatch 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[39202]: pgmap v716: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[39202]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-90"}]: dispatch 2024-03-26T21:32:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-90"}]: dispatch 2024-03-26T21:32:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:38 smithi138 ceph-mon[33706]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T21:32:38.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:38 smithi088 ceph-mon[30075]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T21:32:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:38 smithi088 ceph-mon[39202]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T21:32:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:39 smithi138 ceph-mon[33706]: pgmap v719: 292 pgs: 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:32:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:39 smithi138 ceph-mon[33706]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T21:32:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:39.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:39 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[39202]: pgmap v719: 292 pgs: 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[39202]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[30075]: pgmap v719: 292 pgs: 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[30075]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:39 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:40 smithi138 ceph-mon[33706]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T21:32:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:40.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:40 smithi138 ceph-mon[33706]: pgmap v722: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[39202]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T21:32:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[39202]: pgmap v722: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[30075]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T21:32:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:40.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:40 smithi088 ceph-mon[30075]: pgmap v722: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:32:41.208 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:32:40 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:32:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:41 smithi138 ceph-mon[33706]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T21:32:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:41 smithi088 ceph-mon[30075]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T21:32:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:41 smithi088 ceph-mon[39202]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T21:32:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:42 smithi138 ceph-mon[33706]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T21:32:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-92"}]: dispatch 2024-03-26T21:32:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-92"}]: dispatch 2024-03-26T21:32:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:42 smithi138 ceph-mon[33706]: pgmap v725: 324 pgs: 324 active+clean; 8.4 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:32:42.486 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:32:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:42.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[30075]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T21:32:42.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:42.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:42.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-92"}]: dispatch 2024-03-26T21:32:42.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-92"}]: dispatch 2024-03-26T21:32:42.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[30075]: pgmap v725: 324 pgs: 324 active+clean; 8.4 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:32:42.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[39202]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T21:32:42.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:42.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:32:42.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-92"}]: dispatch 2024-03-26T21:32:42.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-92"}]: dispatch 2024-03-26T21:32:42.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:42 smithi088 ceph-mon[39202]: pgmap v725: 324 pgs: 324 active+clean; 8.4 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T21:32:43.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:43 smithi138 ceph-mon[33706]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T21:32:43.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:43 smithi088 ceph-mon[30075]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T21:32:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:43 smithi088 ceph-mon[39202]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T21:32:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:44 smithi138 ceph-mon[33706]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T21:32:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:44 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:44 smithi138 ceph-mon[33706]: pgmap v728: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:32:44.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[30075]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T21:32:44.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:44.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:44.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[30075]: pgmap v728: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:32:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[39202]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T21:32:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:32:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:44 smithi088 ceph-mon[39202]: pgmap v728: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:32:45.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:45 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:45.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:45 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:45.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:45 smithi138 ceph-mon[33706]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T21:32:45.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[30075]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T21:32:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:45.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:45.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:32:45.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[39202]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T21:32:45.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:32:46.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:46 smithi138 ceph-mon[33706]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T21:32:46.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:46 smithi138 ceph-mon[33706]: pgmap v731: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:32:46.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:46 smithi088 ceph-mon[30075]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T21:32:46.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:46 smithi088 ceph-mon[30075]: pgmap v731: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:32:46.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:46 smithi088 ceph-mon[39202]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T21:32:46.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:46 smithi088 ceph-mon[39202]: pgmap v731: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:32:47.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:47 smithi138 ceph-mon[33706]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T21:32:47.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:47 smithi088 ceph-mon[39202]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T21:32:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:47 smithi088 ceph-mon[30075]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T21:32:48.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:48 smithi138 ceph-mon[33706]: pgmap v733: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-26T21:32:48.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:48 smithi088 ceph-mon[30075]: pgmap v733: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-26T21:32:48.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:48 smithi088 ceph-mon[39202]: pgmap v733: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-26T21:32:50.965 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:50 smithi138 ceph-mon[33706]: pgmap v734: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T21:32:50.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:50 smithi088 ceph-mon[39202]: pgmap v734: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T21:32:50.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:50 smithi088 ceph-mon[30075]: pgmap v734: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T21:32:51.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:32:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:32:51.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:51.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:51.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:52.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:52 smithi088 ceph-mon[39202]: pgmap v735: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:32:52.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:52 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:52 smithi088 ceph-mon[30075]: pgmap v735: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:32:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:52 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:52.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:32:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:32:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:52.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:52 smithi138 ceph-mon[33706]: pgmap v735: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:32:52.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:52 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:32:54.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:54 smithi138 ceph-mon[33706]: pgmap v736: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T21:32:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:54 smithi088 ceph-mon[39202]: pgmap v736: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T21:32:54.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:54 smithi088 ceph-mon[30075]: pgmap v736: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T21:32:55.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:55.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:56.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:56 smithi138 ceph-mon[33706]: pgmap v737: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:32:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:56 smithi088 ceph-mon[39202]: pgmap v737: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:32:56.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:56 smithi088 ceph-mon[30075]: pgmap v737: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:32:57.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:57 smithi138 ceph-mon[33706]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T21:32:57.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:57 smithi088 ceph-mon[30075]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T21:32:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:57 smithi088 ceph-mon[39202]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T21:32:58.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:32:58 smithi138 ceph-mon[33706]: pgmap v739: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:32:58.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:32:58 smithi088 ceph-mon[39202]: pgmap v739: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:32:58.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:32:58 smithi088 ceph-mon[30075]: pgmap v739: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:33:00.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:00 smithi138 ceph-mon[33706]: pgmap v740: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:33:00.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:00 smithi088 ceph-mon[30075]: pgmap v740: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:33:00.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:00 smithi088 ceph-mon[39202]: pgmap v740: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-26T21:33:01.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:33:00 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:33:01.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:01.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:01.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:33:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:33:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:02 smithi138 ceph-mon[33706]: pgmap v741: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:02 smithi138 ceph-mon[33706]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T21:33:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:02 smithi088 ceph-mon[39202]: pgmap v741: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:02 smithi088 ceph-mon[39202]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T21:33:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:02 smithi088 ceph-mon[30075]: pgmap v741: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:02 smithi088 ceph-mon[30075]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T21:33:05.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:04 smithi138 ceph-mon[33706]: pgmap v743: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:04 smithi088 ceph-mon[30075]: pgmap v743: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:04 smithi088 ceph-mon[39202]: pgmap v743: 324 pgs: 324 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:06 smithi138 ceph-mon[33706]: pgmap v744: 324 pgs: 324 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:06 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-94"}]: dispatch 2024-03-26T21:33:07.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:06 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-94"}]: dispatch 2024-03-26T21:33:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[39202]: pgmap v744: 324 pgs: 324 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-94"}]: dispatch 2024-03-26T21:33:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-94"}]: dispatch 2024-03-26T21:33:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[30075]: pgmap v744: 324 pgs: 324 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-94"}]: dispatch 2024-03-26T21:33:07.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:06 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-94"}]: dispatch 2024-03-26T21:33:08.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:07 smithi138 ceph-mon[33706]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T21:33:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:07 smithi088 ceph-mon[39202]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T21:33:08.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:07 smithi088 ceph-mon[30075]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T21:33:09.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:08 smithi138 ceph-mon[33706]: pgmap v746: 292 pgs: 292 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:33:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:08 smithi138 ceph-mon[33706]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T21:33:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:08 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[30075]: pgmap v746: 292 pgs: 292 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:33:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[30075]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T21:33:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[39202]: pgmap v746: 292 pgs: 292 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:33:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[39202]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T21:33:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:08 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:10.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:09 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:09 smithi138 ceph-mon[33706]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T21:33:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:33:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:09 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:09 smithi088 ceph-mon[30075]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T21:33:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:33:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:09 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:09 smithi088 ceph-mon[39202]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T21:33:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:33:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[39202]: pgmap v749: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[39202]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T21:33:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:33:11.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[30075]: pgmap v749: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:11.134 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[30075]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T21:33:11.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:11.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:33:11.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:33:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:33:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:10 smithi138 ceph-mon[33706]: pgmap v749: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:10 smithi138 ceph-mon[33706]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T21:33:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi138 to 3248M 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: Adjusting osd_memory_target on smithi088 to 3248M 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:33:12.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:33:12.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi138 to 3248M 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: Adjusting osd_memory_target on smithi088 to 3248M 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:33:12.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:33:12.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi138 to 3248M 2024-03-26T21:33:12.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: Adjusting osd_memory_target on smithi088 to 3248M 2024-03-26T21:33:12.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:33:12.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:33:12.238 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:33:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:33:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:13 smithi138 ceph-mon[33706]: pgmap v751: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T21:33:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:13 smithi088 ceph-mon[39202]: pgmap v751: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T21:33:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:13 smithi088 ceph-mon[30075]: pgmap v751: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T21:33:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:15 smithi138 ceph-mon[33706]: pgmap v752: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:33:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:15 smithi088 ceph-mon[30075]: pgmap v752: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:33:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:15 smithi088 ceph-mon[39202]: pgmap v752: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T21:33:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:17 smithi138 ceph-mon[33706]: pgmap v753: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 792 B/s wr, 2 op/s 2024-03-26T21:33:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:17 smithi088 ceph-mon[30075]: pgmap v753: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 792 B/s wr, 2 op/s 2024-03-26T21:33:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:17 smithi088 ceph-mon[39202]: pgmap v753: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 792 B/s wr, 2 op/s 2024-03-26T21:33:19.171 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:33:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=cleanup t=2024-03-26T21:33:18.919199225Z level=info msg="Completed cleanup jobs" duration=3.326468ms 2024-03-26T21:33:19.171 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:33:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=sqlstore.transactions t=2024-03-26T21:33:18.92671622Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T21:33:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:19 smithi138 ceph-mon[33706]: pgmap v754: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 701 B/s wr, 2 op/s 2024-03-26T21:33:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:19 smithi088 ceph-mon[39202]: pgmap v754: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 701 B/s wr, 2 op/s 2024-03-26T21:33:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:19 smithi088 ceph-mon[30075]: pgmap v754: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 701 B/s wr, 2 op/s 2024-03-26T21:33:20.177 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:33:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:33:19.916+0000 7fbe23fb9700 -1 snap_mapper.add_oid found existing snaps mapped on 101:8d48e108:test-rados-api-smithi088-79477-97::foo:21, removing 2024-03-26T21:33:20.178 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:33:19 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3[62192]: 2024-03-26T21:33:19.915+0000 7f65886e0700 -1 snap_mapper.add_oid found existing snaps mapped on 101:8d48e108:test-rados-api-smithi088-79477-97::foo:21, removing 2024-03-26T21:33:20.178 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:33:19 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2[56021]: 2024-03-26T21:33:19.915+0000 7f0c1b2bc700 -1 snap_mapper.add_oid found existing snaps mapped on 101:8d48e108:test-rados-api-smithi088-79477-97::foo:21, removing 2024-03-26T21:33:20.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-96"}]: dispatch 2024-03-26T21:33:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:20 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-96"}]: dispatch 2024-03-26T21:33:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-96"}]: dispatch 2024-03-26T21:33:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-96"}]: dispatch 2024-03-26T21:33:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:20.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:20.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-96"}]: dispatch 2024-03-26T21:33:20.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:20 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-96"}]: dispatch 2024-03-26T21:33:21.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:33:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:33:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:21 smithi138 ceph-mon[33706]: pgmap v755: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-03-26T21:33:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:21 smithi138 ceph-mon[33706]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T21:33:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:21 smithi088 ceph-mon[39202]: pgmap v755: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-03-26T21:33:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:21 smithi088 ceph-mon[39202]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T21:33:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:21 smithi088 ceph-mon[30075]: pgmap v755: 324 pgs: 324 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-03-26T21:33:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:21 smithi088 ceph-mon[30075]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T21:33:22.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[39202]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T21:33:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[30075]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T21:33:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:22 smithi138 ceph-mon[33706]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T21:33:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:22 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:33:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:33:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:23 smithi138 ceph-mon[33706]: pgmap v758: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-03-26T21:33:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:23 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:23 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:23 smithi138 ceph-mon[33706]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T21:33:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:23 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[39202]: pgmap v758: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-03-26T21:33:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[39202]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T21:33:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[30075]: pgmap v758: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-03-26T21:33:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[30075]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T21:33:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:23 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:24.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:24 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:33:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:24 smithi138 ceph-mon[33706]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T21:33:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]: dispatch 2024-03-26T21:33:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:24 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]: dispatch 2024-03-26T21:33:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:33:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[30075]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T21:33:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]: dispatch 2024-03-26T21:33:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]: dispatch 2024-03-26T21:33:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:33:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[39202]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T21:33:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]: dispatch 2024-03-26T21:33:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:24 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]: dispatch 2024-03-26T21:33:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:25 smithi138 ceph-mon[33706]: pgmap v761: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T21:33:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:25 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:33:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]': finished 2024-03-26T21:33:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:25 smithi138 ceph-mon[33706]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T21:33:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:25 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[39202]: pgmap v761: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T21:33:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:33:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]': finished 2024-03-26T21:33:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[39202]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T21:33:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[30075]: pgmap v761: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T21:33:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:33:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-98", "mode": "writeback"}]': finished 2024-03-26T21:33:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[30075]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T21:33:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:25 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:26 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]': finished 2024-03-26T21:33:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:26 smithi138 ceph-mon[33706]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T21:33:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:33:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:26 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:33:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:26 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:26 smithi138 ceph-mon[33706]: pgmap v764: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:33:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]': finished 2024-03-26T21:33:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[39202]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T21:33:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:33:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[39202]: pgmap v764: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-98"}]': finished 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[30075]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:26 smithi088 ceph-mon[30075]: pgmap v764: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:33:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:27 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:33:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:33:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:27 smithi138 ceph-mon[33706]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T21:33:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T21:33:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:27 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T21:33:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:33:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:33:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[39202]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T21:33:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T21:33:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T21:33:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:33:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:33:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[30075]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T21:33:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T21:33:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:27 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T21:33:28.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T21:33:28.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:28 smithi138 ceph-mon[33706]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T21:33:28.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:28.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:28 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:28.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:28 smithi138 ceph-mon[33706]: pgmap v767: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:33:28.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T21:33:28.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[39202]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T21:33:28.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:28.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:28.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[39202]: pgmap v767: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:33:28.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T21:33:28.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[30075]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T21:33:28.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:28.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:28.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:28 smithi088 ceph-mon[30075]: pgmap v767: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:33:29.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:33:29.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:29 smithi138 ceph-mon[33706]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T21:33:29.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T21:33:29.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:29 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T21:33:29.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:33:29.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[30075]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T21:33:29.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T21:33:29.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T21:33:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:33:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[39202]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T21:33:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T21:33:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:29 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T21:33:30.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:30 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T21:33:30.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:30 smithi138 ceph-mon[33706]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T21:33:30.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:30.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:30 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:30.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:30 smithi138 ceph-mon[33706]: pgmap v770: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:30.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T21:33:30.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[39202]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T21:33:30.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:30.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:30.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[39202]: pgmap v770: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T21:33:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[30075]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T21:33:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:30.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:30 smithi088 ceph-mon[30075]: pgmap v770: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:31.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:33:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:33:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:33:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:31 smithi138 ceph-mon[33706]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T21:33:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:31 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:33:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[30075]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T21:33:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:33:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[39202]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T21:33:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]: dispatch 2024-03-26T21:33:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:32.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:32 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]': finished 2024-03-26T21:33:32.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:32 smithi138 ceph-mon[33706]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T21:33:32.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:32 smithi138 ceph-mon[33706]: pgmap v773: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:32.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:32 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:32.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]': finished 2024-03-26T21:33:32.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[30075]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T21:33:32.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[30075]: pgmap v773: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:32.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-98"}]': finished 2024-03-26T21:33:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[39202]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T21:33:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[39202]: pgmap v773: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:32.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:32.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:33:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:33:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:33.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:33 smithi138 ceph-mon[33706]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T21:33:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:33 smithi088 ceph-mon[30075]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T21:33:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:33 smithi088 ceph-mon[39202]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T21:33:34.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:34 smithi138 ceph-mon[33706]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T21:33:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:34 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:34.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:34 smithi138 ceph-mon[33706]: pgmap v776: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[39202]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T21:33:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[39202]: pgmap v776: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[30075]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T21:33:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:34.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:34 smithi088 ceph-mon[30075]: pgmap v776: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:35 smithi138 ceph-mon[33706]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T21:33:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:35 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[39202]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T21:33:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[30075]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T21:33:35.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:35.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:35 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:33:36.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: pgmap v779: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]': finished 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]: dispatch 2024-03-26T21:33:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:36 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]: dispatch 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: pgmap v779: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]': finished 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T21:33:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]: dispatch 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]: dispatch 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: pgmap v779: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-6", "overlaypool": "test-rados-api-smithi088-79477-100"}]': finished 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]: dispatch 2024-03-26T21:33:36.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:36 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]: dispatch 2024-03-26T21:33:37.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:37 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:33:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]': finished 2024-03-26T21:33:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:37 smithi138 ceph-mon[33706]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T21:33:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:33:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:37 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:33:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:33:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]': finished 2024-03-26T21:33:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[30075]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T21:33:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:33:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:33:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:33:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-100", "mode": "writeback"}]': finished 2024-03-26T21:33:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[39202]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T21:33:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:33:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:37 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:33:38.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:38 smithi138 ceph-mon[33706]: pgmap v782: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:38.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:33:38.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:38 smithi138 ceph-mon[33706]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T21:33:38.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:38.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:38 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:38.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[39202]: pgmap v782: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:38.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:33:38.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[39202]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T21:33:38.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:38.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:38.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[30075]: pgmap v782: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:38.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:33:38.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[30075]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T21:33:38.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:38.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:38 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:33:40.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:33:40.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:40 smithi138 ceph-mon[33706]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T21:33:40.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:33:40.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:40 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:33:40.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:40 smithi138 ceph-mon[33706]: pgmap v785: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:40.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:33:40.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[39202]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T21:33:40.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:33:40.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:33:40.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[39202]: pgmap v785: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:40.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:33:40.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[30075]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T21:33:40.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:33:40.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:33:40.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:40 smithi088 ceph-mon[30075]: pgmap v785: 324 pgs: 324 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:41.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:33:41.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:33:41.635 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T21:33:41.636 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T21:33:41.636 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T21:33:41.636 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:41.636 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:41.636 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:33:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:33:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T21:33:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T21:33:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T21:33:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:33:41.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:41.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T21:33:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:42 smithi138 ceph-mon[33706]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T21:33:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:42 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:42 smithi138 ceph-mon[33706]: pgmap v788: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T21:33:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[30075]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T21:33:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[30075]: pgmap v788: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:33:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T21:33:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[39202]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T21:33:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:42 smithi088 ceph-mon[39202]: pgmap v788: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:43.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:33:43.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:43 smithi138 ceph-mon[33706]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T21:33:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:43 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:33:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[39202]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T21:33:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]': finished 2024-03-26T21:33:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[30075]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T21:33:43.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:43.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:43 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]: dispatch 2024-03-26T21:33:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:44 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]': finished 2024-03-26T21:33:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:44 smithi138 ceph-mon[33706]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T21:33:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:44 smithi138 ceph-mon[33706]: pgmap v791: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:44.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:44 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]': finished 2024-03-26T21:33:44.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:44 smithi088 ceph-mon[39202]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T21:33:44.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:44 smithi088 ceph-mon[39202]: pgmap v791: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:44.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:44 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-100"}]': finished 2024-03-26T21:33:44.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:44 smithi088 ceph-mon[30075]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T21:33:44.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:44 smithi088 ceph-mon[30075]: pgmap v791: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:33:45.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:45 smithi138 ceph-mon[33706]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T21:33:45.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:45 smithi088 ceph-mon[39202]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T21:33:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:45 smithi088 ceph-mon[30075]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T21:33:46.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:46 smithi138 ceph-mon[33706]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T21:33:46.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:46.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:46 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:46.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:46 smithi138 ceph-mon[33706]: pgmap v794: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:46.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[30075]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T21:33:46.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:46.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:46.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[30075]: pgmap v794: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:46.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[39202]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T21:33:46.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:46.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:46.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:46 smithi088 ceph-mon[39202]: pgmap v794: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:47.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:47 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:47.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:47.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:47 smithi138 ceph-mon[33706]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T21:33:47.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:33:47.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:33:47.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:47 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:33:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[30075]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T21:33:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:33:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:33:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:33:47.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[39202]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T21:33:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T21:33:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:33:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:47 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: pgmap v797: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:33:48.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:48 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:33:48.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:33:48.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T21:33:48.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:33:48.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:33:48.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: pgmap v797: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: pgmap v797: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:48.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T21:33:48.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T21:33:48.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:33:48.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:48 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T21:33:50.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:50 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:33:50.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:50 smithi138 ceph-mon[33706]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T21:33:50.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:50.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:50 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:50.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-102"}]: dispatch 2024-03-26T21:33:50.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:50 smithi138 ceph-mon[33706]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-102"}]: dispatch 2024-03-26T21:33:50.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:50 smithi138 ceph-mon[33706]: pgmap v800: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:50.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:33:50.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[30075]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T21:33:50.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:50.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:50.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-102"}]: dispatch 2024-03-26T21:33:50.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[30075]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-102"}]: dispatch 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[30075]: pgmap v800: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[39202]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-6"}]: dispatch 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2083073499' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-102"}]: dispatch 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[39202]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-6", "tierpool": "test-rados-api-smithi088-79477-102"}]: dispatch 2024-03-26T21:33:50.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:50 smithi088 ceph-mon[39202]: pgmap v800: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:51.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:33:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:33:51.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:51 smithi138 ceph-mon[33706]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T21:33:51.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:51.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:51 smithi088 ceph-mon[39202]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T21:33:51.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:51 smithi088 ceph-mon[30075]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T21:33:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: pgmap v803: 260 pgs: 260 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:52.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:52 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:33:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:52.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: pgmap v803: 260 pgs: 260 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:33:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:33:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: pgmap v803: 260 pgs: 260 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:52 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:53.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:53 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:33:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:53 smithi138 ceph-mon[33706]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T21:33:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:53 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:33:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[39202]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T21:33:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:33:53.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[30075]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T21:33:53.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:53.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:53 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:33:54.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:54 smithi138 ceph-mon[33706]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T21:33:54.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:54 smithi138 ceph-mon[33706]: pgmap v806: 260 pgs: 260 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:54.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:54 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:33:54.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:54 smithi138 ceph-mon[33706]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[39202]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[39202]: pgmap v806: 260 pgs: 260 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[39202]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[30075]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[30075]: pgmap v806: 260 pgs: 260 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi088-79477-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:33:54.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:54 smithi088 ceph-mon[30075]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T21:33:56.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:56 smithi138 ceph-mon[33706]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T21:33:56.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:56.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:56 smithi138 ceph-mon[33706]: pgmap v809: 268 pgs: 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:56.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:56 smithi088 ceph-mon[39202]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T21:33:56.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:56.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:56 smithi088 ceph-mon[39202]: pgmap v809: 268 pgs: 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:56.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:56 smithi088 ceph-mon[30075]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T21:33:56.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:56.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:56 smithi088 ceph-mon[30075]: pgmap v809: 268 pgs: 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:57.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:57 smithi138 ceph-mon[33706]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T21:33:57.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:57.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:57 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:57 smithi088 ceph-mon[39202]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T21:33:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:57 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:57.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:57 smithi088 ceph-mon[30075]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T21:33:57.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:57.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:57 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:33:58.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:58.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:58 smithi138 ceph-mon[33706]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T21:33:58.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:58 smithi138 ceph-mon[33706]: pgmap v812: 300 pgs: 32 unknown, 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:58.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:58.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:58 smithi088 ceph-mon[30075]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T21:33:58.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:58 smithi088 ceph-mon[30075]: pgmap v812: 300 pgs: 32 unknown, 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:58.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:33:58.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:58 smithi088 ceph-mon[39202]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T21:33:58.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:58 smithi088 ceph-mon[39202]: pgmap v812: 300 pgs: 32 unknown, 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:33:59.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:59 smithi138 ceph-mon[33706]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T21:33:59.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:33:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:59.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:59 smithi088 ceph-mon[39202]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T21:33:59.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:33:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:33:59.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:59 smithi088 ceph-mon[30075]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T21:33:59.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:33:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:00 smithi138 ceph-mon[33706]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T21:34:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:00 smithi138 ceph-mon[33706]: pgmap v815: 332 pgs: 64 unknown, 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:00.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[30075]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T21:34:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[30075]: pgmap v815: 332 pgs: 64 unknown, 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:00.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:00.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[39202]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T21:34:00.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:00.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[39202]: pgmap v815: 332 pgs: 64 unknown, 2 creating+activating, 266 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:00.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:01.413 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:34:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:34:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:01 smithi138 ceph-mon[33706]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T21:34:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-107", "overlaypool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-107", "overlaypool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:01 smithi138 ceph-mon[33706]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T21:34:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:01 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[30075]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-107", "overlaypool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-107", "overlaypool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[30075]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[39202]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T21:34:01.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-107", "overlaypool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:01.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:01.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-107", "overlaypool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:01.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[39202]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T21:34:01.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:01 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:34:02.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:02 smithi138 ceph-mon[33706]: pgmap v818: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:02 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:02 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-107-cache", "mode": "writeback"}]': finished 2024-03-26T21:34:02.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:02 smithi138 ceph-mon[33706]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T21:34:02.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[30075]: pgmap v818: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:02.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:02.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-107-cache", "mode": "writeback"}]': finished 2024-03-26T21:34:02.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[30075]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T21:34:02.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[39202]: pgmap v818: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:02.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:02.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-107-cache", "mode": "writeback"}]': finished 2024-03-26T21:34:02.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-mon[39202]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T21:34:02.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:34:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:34:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:03.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:03 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-107"}]: dispatch 2024-03-26T21:34:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:03 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-107"}]: dispatch 2024-03-26T21:34:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:03 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-107"}]: dispatch 2024-03-26T21:34:04.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-107"}]': finished 2024-03-26T21:34:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:04 smithi138 ceph-mon[33706]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T21:34:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:04.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:04 smithi138 ceph-mon[33706]: pgmap v821: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:04.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-107"}]': finished 2024-03-26T21:34:04.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[30075]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T21:34:04.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:04.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[30075]: pgmap v821: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:04.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-107"}]': finished 2024-03-26T21:34:04.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[39202]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T21:34:04.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]: dispatch 2024-03-26T21:34:04.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:04 smithi088 ceph-mon[39202]: pgmap v821: 332 pgs: 332 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:05.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:05 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:05 smithi138 ceph-mon[33706]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T21:34:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:05 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:05 smithi088 ceph-mon[30075]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T21:34:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:05 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/79072903' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-107", "tierpool": "test-rados-api-smithi088-79477-107-cache"}]': finished 2024-03-26T21:34:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:05 smithi088 ceph-mon[39202]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T21:34:06.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:06 smithi138 ceph-mon[33706]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T21:34:06.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:06 smithi138 ceph-mon[33706]: pgmap v824: 300 pgs: 300 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:06.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:06 smithi088 ceph-mon[39202]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T21:34:06.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:06 smithi088 ceph-mon[39202]: pgmap v824: 300 pgs: 300 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:06 smithi088 ceph-mon[30075]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T21:34:06.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:06 smithi088 ceph-mon[30075]: pgmap v824: 300 pgs: 300 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:07.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:07 smithi138 ceph-mon[33706]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T21:34:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:07 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[30075]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T21:34:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[39202]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T21:34:07.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:07.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:07.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:07 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:34:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:08 smithi138 ceph-mon[33706]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T21:34:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:08 smithi138 ceph-mon[33706]: pgmap v827: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:08.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:34:08.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[39202]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T21:34:08.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:08.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[39202]: pgmap v827: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:34:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[30075]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T21:34:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:08.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:08 smithi088 ceph-mon[30075]: pgmap v827: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:09.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:09 smithi138 ceph-mon[33706]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T21:34:09.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:09 smithi088 ceph-mon[30075]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T21:34:09.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:09 smithi088 ceph-mon[39202]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T21:34:10.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:10 smithi138 ceph-mon[33706]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T21:34:10.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:10 smithi138 ceph-mon[33706]: pgmap v830: 276 pgs: 8 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:10.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:10.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:10 smithi088 ceph-mon[39202]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T21:34:10.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:10 smithi088 ceph-mon[39202]: pgmap v830: 276 pgs: 8 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:10.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:10.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:10 smithi088 ceph-mon[30075]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T21:34:10.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:10 smithi088 ceph-mon[30075]: pgmap v830: 276 pgs: 8 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:11.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:34:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:34:11.884 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:11.884 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[39202]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T21:34:11.884 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:11.884 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:11.884 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:11.884 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:34:11.885 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:11.885 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[30075]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T21:34:11.885 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:11.885 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:11.885 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:11.885 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:34:11.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:11 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:11.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:11 smithi138 ceph-mon[33706]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T21:34:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:11.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:34:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:34:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:12.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:12.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T21:34:12.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:12.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: pgmap v833: 308 pgs: 32 unknown, 8 creating+peering, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-109", "overlaypool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: pgmap v833: 308 pgs: 32 unknown, 8 creating+peering, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:34:12.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:34:12.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:34:12.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:12.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T21:34:12.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-109", "overlaypool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: pgmap v833: 308 pgs: 32 unknown, 8 creating+peering, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T21:34:12.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-109", "overlaypool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-109", "overlaypool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:14 smithi138 ceph-mon[33706]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T21:34:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:34:14.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:14 smithi138 ceph-mon[33706]: pgmap v836: 308 pgs: 32 unknown, 8 creating+peering, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:14.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-109", "overlaypool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:14.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[39202]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T21:34:14.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:34:14.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[39202]: pgmap v836: 308 pgs: 32 unknown, 8 creating+peering, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-109", "overlaypool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[30075]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T21:34:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T21:34:14.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:14 smithi088 ceph-mon[30075]: pgmap v836: 308 pgs: 32 unknown, 8 creating+peering, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:15.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:15 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:15.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-109-cache", "mode": "writeback"}]': finished 2024-03-26T21:34:15.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:15 smithi138 ceph-mon[33706]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T21:34:15.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:34:15.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:15.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-109-cache", "mode": "writeback"}]': finished 2024-03-26T21:34:15.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[30075]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T21:34:15.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:34:15.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:15.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-109-cache", "mode": "writeback"}]': finished 2024-03-26T21:34:15.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[39202]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T21:34:15.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:34:16.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:34:16.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:16 smithi138 ceph-mon[33706]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T21:34:16.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:34:16.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:16 smithi138 ceph-mon[33706]: pgmap v839: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:16.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:34:16.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[39202]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T21:34:16.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:34:16.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[39202]: pgmap v839: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:16.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:34:16.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[30075]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T21:34:16.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:34:16.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:16 smithi088 ceph-mon[30075]: pgmap v839: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:17.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:34:17.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:17 smithi138 ceph-mon[33706]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T21:34:17.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:34:17.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:17 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:17.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:34:17.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:17 smithi138 ceph-mon[33706]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T21:34:17.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:34:17.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[30075]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[30075]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[39202]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:34:17.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:17.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:34:17.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:17 smithi088 ceph-mon[39202]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T21:34:18.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T21:34:18.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:18 smithi138 ceph-mon[33706]: pgmap v842: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:18.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T21:34:18.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:18 smithi138 ceph-mon[33706]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[30075]: pgmap v842: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[30075]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[39202]: pgmap v842: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T21:34:18.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:18 smithi088 ceph-mon[39202]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T21:34:19.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:19.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:19.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:20.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:20 smithi138 ceph-mon[33706]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T21:34:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:20.975 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:20 smithi138 ceph-mon[33706]: pgmap v845: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:20.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:20.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[30075]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T21:34:20.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:20.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[30075]: pgmap v845: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:20.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:20.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[39202]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T21:34:20.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]: dispatch 2024-03-26T21:34:20.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:20 smithi088 ceph-mon[39202]: pgmap v845: 308 pgs: 308 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:21.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:34:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:34:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:21 smithi138 ceph-mon[33706]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T21:34:21.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:21.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:21.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:21 smithi088 ceph-mon[30075]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T21:34:21.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-109", "tierpool": "test-rados-api-smithi088-79477-109-cache"}]': finished 2024-03-26T21:34:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:21 smithi088 ceph-mon[39202]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T21:34:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:22 smithi088 ceph-mon[30075]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T21:34:22.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:22 smithi088 ceph-mon[30075]: pgmap v848: 276 pgs: 276 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:22.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:22 smithi088 ceph-mon[39202]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T21:34:22.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:22 smithi088 ceph-mon[39202]: pgmap v848: 276 pgs: 276 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:22.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:34:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:34:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:22.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:22 smithi138 ceph-mon[33706]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T21:34:22.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:22 smithi138 ceph-mon[33706]: pgmap v848: 276 pgs: 276 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:23.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:23 smithi138 ceph-mon[33706]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T21:34:23.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:23.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:23.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:23 smithi138 ceph-mon[33706]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T21:34:23.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:23.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[30075]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T21:34:23.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:23.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:23.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[30075]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T21:34:23.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:23.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[39202]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T21:34:23.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:23.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:23.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[39202]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T21:34:23.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]: dispatch 2024-03-26T21:34:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:24 smithi138 ceph-mon[33706]: pgmap v851: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:24.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:24 smithi138 ceph-mon[33706]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T21:34:24.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:24 smithi088 ceph-mon[30075]: pgmap v851: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:24.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:24.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:24 smithi088 ceph-mon[30075]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T21:34:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:24 smithi088 ceph-mon[39202]: pgmap v851: 268 pgs: 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1790706615' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-109"}]': finished 2024-03-26T21:34:24.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:24 smithi088 ceph-mon[39202]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T21:34:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:25 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:25 smithi138 ceph-mon[33706]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T21:34:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:25.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:25 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[30075]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:25.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[39202]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T21:34:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:25.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:25 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:26.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:26 smithi138 ceph-mon[33706]: pgmap v854: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:34:26.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:26 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:34:26.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:26 smithi138 ceph-mon[33706]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T21:34:26.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:26.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:26 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:26.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[30075]: pgmap v854: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:34:26.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:34:26.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[30075]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T21:34:26.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:26.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:26.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[39202]: pgmap v854: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:34:26.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:34:26.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[39202]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T21:34:26.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/190896283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:26.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:26 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]: dispatch 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:28.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:28 smithi138 ceph-mon[33706]: pgmap v857: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:34:28.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:34:28.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T21:34:28.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[30075]: pgmap v857: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: from='client.57143 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi088-79477-104"}]': finished 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:28.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:28.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T21:34:28.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:28 smithi088 ceph-mon[39202]: pgmap v857: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:34:29.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:29 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:34:29.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:29.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:29 smithi138 ceph-mon[33706]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T21:34:29.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:29 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:29.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:34:29.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:29.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[39202]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T21:34:29.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:29.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi088-79477-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T21:34:29.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:29.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[30075]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T21:34:29.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:29 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:30 smithi138 ceph-mon[33706]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T21:34:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:30 smithi138 ceph-mon[33706]: pgmap v860: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:30.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:30 smithi138 ceph-mon[33706]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[30075]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[30075]: pgmap v860: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[30075]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[39202]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[39202]: pgmap v860: 260 pgs: 260 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi088-79477-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:30 smithi088 ceph-mon[39202]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T21:34:31.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:34:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:34:31.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:31.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:31 smithi138 ceph-mon[33706]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T21:34:31.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:31.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:31 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[39202]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T21:34:31.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:31.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[30075]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T21:34:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:31.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:31 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:32.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[39202]: pgmap v863: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:32.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:32.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:32.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[39202]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T21:34:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[30075]: pgmap v863: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:32.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-mon[30075]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T21:34:32.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:34:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:34:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:32.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:32 smithi138 ceph-mon[33706]: pgmap v863: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:32.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:32 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:32.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:32 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:32.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:32 smithi138 ceph-mon[33706]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T21:34:33.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:33.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:33.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:33.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:33 smithi138 ceph-mon[33706]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T21:34:33.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:33.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:33.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:33.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:33.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:33.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[30075]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T21:34:33.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:33.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:33.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:33.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:33.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:33.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[39202]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T21:34:33.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:33.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:34.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:34 smithi138 ceph-mon[33706]: pgmap v866: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:34.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:34 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]': finished 2024-03-26T21:34:34.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:34 smithi138 ceph-mon[33706]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T21:34:34.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:34 smithi088 ceph-mon[39202]: pgmap v866: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:34.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:34 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]': finished 2024-03-26T21:34:34.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:34 smithi088 ceph-mon[39202]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T21:34:34.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:34 smithi088 ceph-mon[30075]: pgmap v866: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:34.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:34 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-112"}]': finished 2024-03-26T21:34:34.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:34 smithi088 ceph-mon[30075]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T21:34:35.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:35.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:35 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:35.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:35.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:35 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:35.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:35.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:35 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:36.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:36 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:36.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:36 smithi138 ceph-mon[33706]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T21:34:36.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:36.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:36 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:36.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:36 smithi138 ceph-mon[33706]: pgmap v869: 300 pgs: 300 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:34:36.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:36.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[39202]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T21:34:36.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:36.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:36.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[39202]: pgmap v869: 300 pgs: 300 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:34:36.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:36.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[30075]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T21:34:36.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:36.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]: dispatch 2024-03-26T21:34:36.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:36 smithi088 ceph-mon[30075]: pgmap v869: 300 pgs: 300 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T21:34:37.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:37 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]': finished 2024-03-26T21:34:37.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:37 smithi138 ceph-mon[33706]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T21:34:37.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:37 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:37 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]': finished 2024-03-26T21:34:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:37 smithi088 ceph-mon[39202]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T21:34:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:37 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:37.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:37 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-112"}]': finished 2024-03-26T21:34:37.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:37 smithi088 ceph-mon[30075]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T21:34:37.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:37 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:38.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:38 smithi138 ceph-mon[33706]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T21:34:38.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:38 smithi138 ceph-mon[33706]: pgmap v872: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:38.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:38 smithi088 ceph-mon[30075]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T21:34:38.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:38 smithi088 ceph-mon[30075]: pgmap v872: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:38.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:38 smithi088 ceph-mon[39202]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T21:34:38.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:38 smithi088 ceph-mon[39202]: pgmap v872: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:39.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:39 smithi138 ceph-mon[33706]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T21:34:39.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:39.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:39 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:39.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:39 smithi088 ceph-mon[30075]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T21:34:39.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:39.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:39 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:39.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:39 smithi088 ceph-mon[39202]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T21:34:39.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:39.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:39 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:40 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:40 smithi138 ceph-mon[33706]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T21:34:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:40 smithi138 ceph-mon[33706]: pgmap v875: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:40 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:40.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:40.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:40.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[30075]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T21:34:40.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[30075]: pgmap v875: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[39202]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[39202]: pgmap v875: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:40.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:41.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:34:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:34:41.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:41 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:41.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:41 smithi138 ceph-mon[33706]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T21:34:41.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:41.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:41 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:41.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[39202]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T21:34:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:41.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:41.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:41.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:41.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[30075]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T21:34:41.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:41.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:41.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[30075]: pgmap v877: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-26T21:34:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]': finished 2024-03-26T21:34:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[30075]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T21:34:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]: dispatch 2024-03-26T21:34:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]: dispatch 2024-03-26T21:34:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:34:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[39202]: pgmap v877: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-26T21:34:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]': finished 2024-03-26T21:34:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[39202]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T21:34:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]: dispatch 2024-03-26T21:34:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:42 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]: dispatch 2024-03-26T21:34:42.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:42 smithi138 ceph-mon[33706]: pgmap v877: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-26T21:34:42.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:42 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-114"}]': finished 2024-03-26T21:34:42.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:42 smithi138 ceph-mon[33706]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T21:34:42.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:42 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]: dispatch 2024-03-26T21:34:42.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:42 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]: dispatch 2024-03-26T21:34:43.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:43 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:43.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:43 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]': finished 2024-03-26T21:34:43.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:43 smithi138 ceph-mon[33706]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T21:34:43.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:43.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:43 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:43.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:43.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]': finished 2024-03-26T21:34:43.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[30075]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T21:34:43.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:43.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:43.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:43.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-114", "mode": "writeback"}]': finished 2024-03-26T21:34:43.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[39202]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T21:34:43.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:43.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:43 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:34:44.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:44 smithi138 ceph-mon[33706]: pgmap v880: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:44.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:44 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:44.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:44 smithi138 ceph-mon[33706]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T21:34:44.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:44.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:44 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[39202]: pgmap v880: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[39202]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T21:34:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[30075]: pgmap v880: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:34:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:34:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[30075]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T21:34:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:44.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:44 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]: dispatch 2024-03-26T21:34:45.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:45 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:45.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:45 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]': finished 2024-03-26T21:34:45.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:45 smithi138 ceph-mon[33706]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T21:34:45.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:45 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:45.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:45 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]': finished 2024-03-26T21:34:45.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:45 smithi088 ceph-mon[39202]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T21:34:45.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:45 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:34:45.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:45 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-114"}]': finished 2024-03-26T21:34:45.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:45 smithi088 ceph-mon[30075]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T21:34:46.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:46 smithi138 ceph-mon[33706]: pgmap v883: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:34:46.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:46 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:46.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:46 smithi138 ceph-mon[33706]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T21:34:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:46 smithi088 ceph-mon[39202]: pgmap v883: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:34:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:46 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:46 smithi088 ceph-mon[39202]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T21:34:46.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:46 smithi088 ceph-mon[30075]: pgmap v883: 300 pgs: 300 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T21:34:46.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:46 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:46.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:46 smithi088 ceph-mon[30075]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T21:34:47.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:47 smithi138 ceph-mon[33706]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T21:34:47.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:47.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:47 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:47.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:47 smithi088 ceph-mon[39202]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T21:34:47.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:47.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:47 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:47.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:47 smithi088 ceph-mon[30075]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T21:34:47.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:47.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:47 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:34:48.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:48 smithi138 ceph-mon[33706]: pgmap v886: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:48.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:48.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:48 smithi138 ceph-mon[33706]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T21:34:48.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:48 smithi088 ceph-mon[30075]: pgmap v886: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:48.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:48.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:48 smithi088 ceph-mon[30075]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T21:34:48.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:48 smithi088 ceph-mon[39202]: pgmap v886: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:48.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:34:48.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:48 smithi088 ceph-mon[39202]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T21:34:49.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:49 smithi088 ceph-mon[30075]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T21:34:49.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:49.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:49 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:49.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:49 smithi088 ceph-mon[39202]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T21:34:49.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:49.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:49 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:50.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:49 smithi138 ceph-mon[33706]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T21:34:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:49 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:34:50.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[30075]: pgmap v889: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:50.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:50.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[30075]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T21:34:50.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:34:50.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:34:50.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[39202]: pgmap v889: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:50.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:50.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[39202]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T21:34:50.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:34:50.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:50 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:50 smithi138 ceph-mon[33706]: pgmap v889: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:50 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:50 smithi138 ceph-mon[33706]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T21:34:51.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:34:51.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:50 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:34:51.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:34:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:34:52.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:51 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]': finished 2024-03-26T21:34:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:51 smithi138 ceph-mon[33706]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T21:34:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]: dispatch 2024-03-26T21:34:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:51 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]: dispatch 2024-03-26T21:34:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]': finished 2024-03-26T21:34:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[30075]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T21:34:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]: dispatch 2024-03-26T21:34:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]: dispatch 2024-03-26T21:34:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-116"}]': finished 2024-03-26T21:34:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[39202]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T21:34:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]: dispatch 2024-03-26T21:34:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]: dispatch 2024-03-26T21:34:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:34:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:52.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: pgmap v892: 300 pgs: 300 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:34:52.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:52.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]': finished 2024-03-26T21:34:52.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T21:34:52.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "309.1"}]: dispatch 2024-03-26T21:34:52.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "309.1"}]: dispatch 2024-03-26T21:34:53.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: pgmap v892: 300 pgs: 300 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]': finished 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "309.1"}]: dispatch 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "309.1"}]: dispatch 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: 309.1 scrub starts 2024-03-26T21:34:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:52 smithi138 ceph-mon[33706]: 309.1 scrub ok 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: 309.1 scrub starts 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[30075]: 309.1 scrub ok 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: pgmap v892: 300 pgs: 300 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-116", "mode": "writeback"}]': finished 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "309.1"}]: dispatch 2024-03-26T21:34:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: from='mon.? v1:172.21.15.88:0/2356015521' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "309.1"}]: dispatch 2024-03-26T21:34:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:34:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: 309.1 scrub starts 2024-03-26T21:34:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:52 smithi088 ceph-mon[39202]: 309.1 scrub ok 2024-03-26T21:34:55.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:54 smithi138 ceph-mon[33706]: pgmap v894: 300 pgs: 300 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-26T21:34:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:54 smithi088 ceph-mon[39202]: pgmap v894: 300 pgs: 300 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-26T21:34:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:54 smithi088 ceph-mon[30075]: pgmap v894: 300 pgs: 300 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-26T21:34:56.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:56 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:34:56.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:56.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:56 smithi138 ceph-mon[33706]: pgmap v895: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:34:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:56 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:34:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:56 smithi088 ceph-mon[39202]: pgmap v895: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:34:56.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:56 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:34:56.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:56.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:56 smithi088 ceph-mon[30075]: pgmap v895: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:34:58.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:34:58 smithi138 ceph-mon[33706]: pgmap v896: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 904 B/s wr, 2 op/s 2024-03-26T21:34:58.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:34:58 smithi088 ceph-mon[30075]: pgmap v896: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 904 B/s wr, 2 op/s 2024-03-26T21:34:58.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:34:58 smithi088 ceph-mon[39202]: pgmap v896: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 904 B/s wr, 2 op/s 2024-03-26T21:35:00.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:00 smithi138 ceph-mon[33706]: pgmap v897: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 1 op/s 2024-03-26T21:35:00.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:00 smithi088 ceph-mon[39202]: pgmap v897: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 1 op/s 2024-03-26T21:35:00.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:00 smithi088 ceph-mon[30075]: pgmap v897: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 1 op/s 2024-03-26T21:35:01.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:35:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:35:01.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:01.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:01.984 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:02.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:02 smithi088 ceph-mon[30075]: pgmap v898: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:02.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:02 smithi088 ceph-mon[39202]: pgmap v898: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:35:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:35:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:02.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:02 smithi138 ceph-mon[33706]: pgmap v898: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:04.973 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:04 smithi138 ceph-mon[33706]: pgmap v899: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:35:04.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:04 smithi088 ceph-mon[39202]: pgmap v899: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:35:04.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:04 smithi088 ceph-mon[30075]: pgmap v899: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:35:06.974 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:06 smithi138 ceph-mon[33706]: pgmap v900: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:35:06.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:06 smithi088 ceph-mon[30075]: pgmap v900: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:35:06.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:06 smithi088 ceph-mon[39202]: pgmap v900: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:35:09.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:08 smithi138 ceph-mon[33706]: pgmap v901: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:35:09.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:08 smithi138 ceph-mon[33706]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T21:35:09.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:09.226 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:08 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[39202]: pgmap v901: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:35:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[39202]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T21:35:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[30075]: pgmap v901: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:35:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[30075]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T21:35:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:09.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:08 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:10.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:09 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:09 smithi138 ceph-mon[33706]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T21:35:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:35:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:09 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:35:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:09 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:09 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]': finished 2024-03-26T21:35:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:09 smithi138 ceph-mon[33706]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T21:35:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[30075]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T21:35:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:35:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:35:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]': finished 2024-03-26T21:35:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[30075]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T21:35:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[39202]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T21:35:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:35:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]: dispatch 2024-03-26T21:35:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-116"}]': finished 2024-03-26T21:35:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:09 smithi088 ceph-mon[39202]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T21:35:11.100 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:10 smithi138 ceph-mon[33706]: pgmap v904: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:35:11.100 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:11.100 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:10 smithi088 ceph-mon[39202]: pgmap v904: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:35:11.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:11.135 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:11.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:10 smithi088 ceph-mon[30075]: pgmap v904: 300 pgs: 300 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:35:11.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:11.135 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:11.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:35:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:35:12.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:11 smithi138 ceph-mon[33706]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T21:35:12.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:11 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:11 smithi088 ceph-mon[39202]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T21:35:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:11 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:11 smithi088 ceph-mon[30075]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T21:35:12.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:11 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:35:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:13.050 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[39202]: pgmap v907: 268 pgs: 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[39202]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[39202]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[30075]: pgmap v907: 268 pgs: 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[30075]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T21:35:13.051 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:13.052 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:13.052 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:13.052 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[30075]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T21:35:13.052 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:35:13.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:12 smithi138 ceph-mon[33706]: pgmap v907: 268 pgs: 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T21:35:13.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:12 smithi138 ceph-mon[33706]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T21:35:13.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:13.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:12 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:13.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:12 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:13.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:12 smithi138 ceph-mon[33706]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T21:35:13.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:35:14.063 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:13 smithi138 ceph-mon[33706]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T21:35:14.063 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:14.063 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:13 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:14.206 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:13 smithi088 ceph-mon[39202]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T21:35:14.206 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:14.206 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:13 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:14.206 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:13 smithi088 ceph-mon[30075]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T21:35:14.206 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:14.206 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:13 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:14 smithi138 ceph-mon[33706]: pgmap v911: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 783 B/s rd, 1 op/s 2024-03-26T21:35:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:14 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:14 smithi138 ceph-mon[33706]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T21:35:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:15.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:14 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[39202]: pgmap v911: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 783 B/s rd, 1 op/s 2024-03-26T21:35:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[39202]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T21:35:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[30075]: pgmap v911: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 783 B/s rd, 1 op/s 2024-03-26T21:35:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[30075]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T21:35:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:15.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:14 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]': finished 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]: dispatch 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]: dispatch 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]': finished 2024-03-26T21:35:16.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:15 smithi138 ceph-mon[33706]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]': finished 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]: dispatch 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]: dispatch 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:16.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]': finished 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[30075]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-118"}]': finished 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]: dispatch 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]: dispatch 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-118", "mode": "writeback"}]': finished 2024-03-26T21:35:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:15 smithi088 ceph-mon[39202]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T21:35:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:17 smithi138 ceph-mon[33706]: pgmap v914: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 527 B/s wr, 1 op/s 2024-03-26T21:35:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:17 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:17 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:17 smithi138 ceph-mon[33706]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T21:35:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:17 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[30075]: pgmap v914: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 527 B/s wr, 1 op/s 2024-03-26T21:35:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[30075]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T21:35:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[39202]: pgmap v914: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 527 B/s wr, 1 op/s 2024-03-26T21:35:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[39202]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T21:35:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:17.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:17 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:18 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:35:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:18 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:35:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:18 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:18 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:18 smithi138 ceph-mon[33706]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T21:35:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:18 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:35:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:35:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[39202]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T21:35:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[30075]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:18 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]: dispatch 2024-03-26T21:35:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:19 smithi138 ceph-mon[33706]: pgmap v917: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:35:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:19 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:19 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]': finished 2024-03-26T21:35:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:19 smithi138 ceph-mon[33706]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T21:35:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[39202]: pgmap v917: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:35:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]': finished 2024-03-26T21:35:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[39202]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T21:35:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[30075]: pgmap v917: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:35:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-118"}]': finished 2024-03-26T21:35:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:19 smithi088 ceph-mon[30075]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T21:35:21.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:20 smithi138 ceph-mon[33706]: pgmap v920: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:20 smithi138 ceph-mon[33706]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T21:35:21.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:20 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:21.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:35:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:35:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:20 smithi088 ceph-mon[39202]: pgmap v920: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:20 smithi088 ceph-mon[39202]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T21:35:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:20 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:20 smithi088 ceph-mon[30075]: pgmap v920: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:20 smithi088 ceph-mon[30075]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T21:35:21.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:20 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:21 smithi138 ceph-mon[33706]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T21:35:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:21 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:21 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[39202]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[30075]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:21 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:35:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:23.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:22 smithi138 ceph-mon[33706]: pgmap v923: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:22 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:22 smithi138 ceph-mon[33706]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T21:35:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:22 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[39202]: pgmap v923: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[39202]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T21:35:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[30075]: pgmap v923: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[30075]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T21:35:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:23.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:22 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:23 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:23 smithi138 ceph-mon[33706]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T21:35:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:24.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:23 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[39202]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[30075]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:24.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:23 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:25.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:24 smithi138 ceph-mon[33706]: pgmap v926: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:24 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]': finished 2024-03-26T21:35:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:24 smithi138 ceph-mon[33706]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T21:35:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]: dispatch 2024-03-26T21:35:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:24 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]: dispatch 2024-03-26T21:35:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[39202]: pgmap v926: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]': finished 2024-03-26T21:35:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[39202]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T21:35:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]: dispatch 2024-03-26T21:35:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]: dispatch 2024-03-26T21:35:25.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[30075]: pgmap v926: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:25.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-120"}]': finished 2024-03-26T21:35:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[30075]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T21:35:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]: dispatch 2024-03-26T21:35:25.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:24 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]: dispatch 2024-03-26T21:35:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:25 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]': finished 2024-03-26T21:35:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:25 smithi138 ceph-mon[33706]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T21:35:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:25 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:26.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]': finished 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[30075]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-120", "mode": "writeback"}]': finished 2024-03-26T21:35:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[39202]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T21:35:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:27.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: pgmap v929: 300 pgs: 300 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]': finished 2024-03-26T21:35:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:26 smithi138 ceph-mon[33706]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: pgmap v929: 300 pgs: 300 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]': finished 2024-03-26T21:35:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[30075]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: pgmap v929: 300 pgs: 300 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]: dispatch 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-120"}]': finished 2024-03-26T21:35:27.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:26 smithi088 ceph-mon[39202]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T21:35:29.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:28 smithi138 ceph-mon[33706]: pgmap v932: 300 pgs: 300 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T21:35:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:28 smithi138 ceph-mon[33706]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T21:35:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:28 smithi088 ceph-mon[30075]: pgmap v932: 300 pgs: 300 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T21:35:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:28 smithi088 ceph-mon[30075]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T21:35:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:28 smithi088 ceph-mon[39202]: pgmap v932: 300 pgs: 300 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T21:35:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:28 smithi088 ceph-mon[39202]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T21:35:30.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:29 smithi138 ceph-mon[33706]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T21:35:30.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:29 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:30.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:29 smithi088 ceph-mon[30075]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T21:35:30.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:30.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:29 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:29 smithi088 ceph-mon[39202]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T21:35:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:29 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:31.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:35:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:35:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:30 smithi138 ceph-mon[33706]: pgmap v935: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:30 smithi138 ceph-mon[33706]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T21:35:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:31.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[39202]: pgmap v935: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[39202]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T21:35:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[30075]: pgmap v935: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[30075]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T21:35:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:32.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:31 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:31 smithi138 ceph-mon[33706]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T21:35:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:31 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:31 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[39202]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T21:35:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[30075]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:31 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:35:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:33.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:32 smithi138 ceph-mon[33706]: pgmap v938: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:35:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:32 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]': finished 2024-03-26T21:35:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:32 smithi138 ceph-mon[33706]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T21:35:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]: dispatch 2024-03-26T21:35:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:32 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]: dispatch 2024-03-26T21:35:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[30075]: pgmap v938: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:35:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]': finished 2024-03-26T21:35:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[30075]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T21:35:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]: dispatch 2024-03-26T21:35:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]: dispatch 2024-03-26T21:35:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[39202]: pgmap v938: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:35:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-122"}]': finished 2024-03-26T21:35:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[39202]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T21:35:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]: dispatch 2024-03-26T21:35:33.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:32 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]: dispatch 2024-03-26T21:35:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:33 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]': finished 2024-03-26T21:35:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:33 smithi138 ceph-mon[33706]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T21:35:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]': finished 2024-03-26T21:35:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[39202]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T21:35:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-122", "mode": "writeback"}]': finished 2024-03-26T21:35:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[30075]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T21:35:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:35.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:34 smithi138 ceph-mon[33706]: pgmap v941: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:34 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:34 smithi138 ceph-mon[33706]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T21:35:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:34 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[39202]: pgmap v941: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[39202]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[30075]: pgmap v941: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[30075]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:34 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]: dispatch 2024-03-26T21:35:36.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:35 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:35 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]': finished 2024-03-26T21:35:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:35 smithi138 ceph-mon[33706]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T21:35:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:35 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:35 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]': finished 2024-03-26T21:35:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:35 smithi088 ceph-mon[39202]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T21:35:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:35 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:35 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-122"}]': finished 2024-03-26T21:35:36.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:35 smithi088 ceph-mon[30075]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T21:35:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:36 smithi138 ceph-mon[33706]: pgmap v944: 300 pgs: 300 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-26T21:35:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:36 smithi138 ceph-mon[33706]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T21:35:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:36 smithi088 ceph-mon[39202]: pgmap v944: 300 pgs: 300 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-26T21:35:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:36 smithi088 ceph-mon[39202]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T21:35:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:36 smithi088 ceph-mon[30075]: pgmap v944: 300 pgs: 300 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-26T21:35:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:36 smithi088 ceph-mon[30075]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T21:35:38.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:37 smithi138 ceph-mon[33706]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T21:35:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:37 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:38.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:37 smithi088 ceph-mon[39202]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T21:35:38.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:38.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:37 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:37 smithi088 ceph-mon[30075]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T21:35:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:37 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:38 smithi138 ceph-mon[33706]: pgmap v947: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:38 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:38 smithi138 ceph-mon[33706]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T21:35:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:38 smithi088 ceph-mon[39202]: pgmap v947: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:38 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:39.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:38 smithi088 ceph-mon[39202]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T21:35:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:38 smithi088 ceph-mon[30075]: pgmap v947: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:38 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:38 smithi088 ceph-mon[30075]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T21:35:40.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:39 smithi138 ceph-mon[33706]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T21:35:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:39 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:39 smithi088 ceph-mon[39202]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T21:35:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:39 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:39 smithi088 ceph-mon[30075]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T21:35:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:40.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:39 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:41.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: pgmap v950: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: pgmap v950: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: pgmap v950: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T21:35:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:35:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]': finished 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]: dispatch 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]: dispatch 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]': finished 2024-03-26T21:35:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[39202]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]': finished 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]: dispatch 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]: dispatch 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]': finished 2024-03-26T21:35:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:41 smithi088 ceph-mon[30075]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T21:35:42.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-124"}]': finished 2024-03-26T21:35:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T21:35:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]: dispatch 2024-03-26T21:35:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]: dispatch 2024-03-26T21:35:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-124", "mode": "writeback"}]': finished 2024-03-26T21:35:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:41 smithi138 ceph-mon[33706]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T21:35:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:35:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:35:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:42 smithi088 ceph-mon[39202]: pgmap v953: 300 pgs: 300 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:35:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:42 smithi088 ceph-mon[30075]: pgmap v953: 300 pgs: 300 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:35:43.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:42 smithi138 ceph-mon[33706]: pgmap v953: 300 pgs: 300 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:35:44.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:44 smithi138 ceph-mon[33706]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T21:35:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:44 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:44 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:44 smithi088 ceph-mon[39202]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T21:35:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:44 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:44 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:44.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:43 smithi088 ceph-mon[30075]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T21:35:44.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:44 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:44.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:44 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:45 smithi138 ceph-mon[33706]: pgmap v956: 300 pgs: 300 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:35:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:45 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:45 smithi138 ceph-mon[33706]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T21:35:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:45 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[30075]: pgmap v956: 300 pgs: 300 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:35:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[30075]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T21:35:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[39202]: pgmap v956: 300 pgs: 300 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T21:35:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[39202]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T21:35:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:45 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]: dispatch 2024-03-26T21:35:46.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:46 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:46 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]': finished 2024-03-26T21:35:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:46 smithi138 ceph-mon[33706]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T21:35:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:46 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:46 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]': finished 2024-03-26T21:35:46.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:46 smithi088 ceph-mon[39202]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T21:35:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:46 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:46 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-124"}]': finished 2024-03-26T21:35:46.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:46 smithi088 ceph-mon[30075]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T21:35:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:47 smithi138 ceph-mon[33706]: pgmap v959: 300 pgs: 300 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-26T21:35:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:47 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:47 smithi138 ceph-mon[33706]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T21:35:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:47 smithi138 ceph-mon[33706]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T21:35:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:47 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[30075]: pgmap v959: 300 pgs: 300 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[30075]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[30075]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[39202]: pgmap v959: 300 pgs: 300 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[39202]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T21:35:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[39202]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T21:35:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:47 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:49.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:48 smithi138 ceph-mon[33706]: pgmap v962: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:48 smithi138 ceph-mon[33706]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T21:35:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[39202]: pgmap v962: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[39202]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[30075]: pgmap v962: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[30075]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T21:35:49.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:49.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:50.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:49 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:49 smithi138 ceph-mon[33706]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T21:35:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:49 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:49 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]': finished 2024-03-26T21:35:50.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:49 smithi138 ceph-mon[33706]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[39202]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]': finished 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[39202]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:50.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[30075]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T21:35:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-126"}]': finished 2024-03-26T21:35:50.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:49 smithi088 ceph-mon[30075]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T21:35:51.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:35:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:35:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:50 smithi138 ceph-mon[33706]: pgmap v965: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]: dispatch 2024-03-26T21:35:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:50 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]: dispatch 2024-03-26T21:35:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:50 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:50 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]': finished 2024-03-26T21:35:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:50 smithi138 ceph-mon[33706]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T21:35:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[39202]: pgmap v965: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]: dispatch 2024-03-26T21:35:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]: dispatch 2024-03-26T21:35:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]': finished 2024-03-26T21:35:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[39202]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T21:35:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[30075]: pgmap v965: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:35:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]: dispatch 2024-03-26T21:35:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]: dispatch 2024-03-26T21:35:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:35:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-126", "mode": "writeback"}]': finished 2024-03-26T21:35:51.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:50 smithi088 ceph-mon[30075]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T21:35:52.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:51 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:51 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:51 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:51 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:35:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:53.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:52 smithi138 ceph-mon[33706]: pgmap v968: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:52 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:52 smithi138 ceph-mon[33706]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T21:35:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:52 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[39202]: pgmap v968: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[39202]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T21:35:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[30075]: pgmap v968: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:53.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:35:53.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[30075]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T21:35:53.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:53.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:52 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]: dispatch 2024-03-26T21:35:53.917 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:35:53.917 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9051 ms) 2024-03-26T21:35:53.917 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-26T21:35:53.917 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:35:53.917 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6048 ms) 2024-03-26T21:35:53.917 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-26T21:35:53.917 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5046 ms) 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (24630 ms) 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13326 ms) 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12088 ms) 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 00000012 2024-03-26T21:35:53.918 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12053 ms) 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6030 ms) 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (552603 ms total) 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1034 ms) 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11040 ms) 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-26T21:35:53.919 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18102 ms) 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (1 ms) 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30178 ms total) 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7057 ms) 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8082 ms) 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: waiting for scrub... 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: done waiting 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (25052 ms) 2024-03-26T21:35:53.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9138 ms) 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7979 ms) 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8061 ms) 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10074 ms) 2024-03-26T21:35:53.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-26T21:35:54.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:53 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:53 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]': finished 2024-03-26T21:35:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:53 smithi138 ceph-mon[33706]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T21:35:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:53 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:53 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]': finished 2024-03-26T21:35:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:53 smithi088 ceph-mon[39202]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T21:35:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:53 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:35:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:53 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-126"}]': finished 2024-03-26T21:35:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:53 smithi088 ceph-mon[30075]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T21:35:55.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:54 smithi138 ceph-mon[33706]: pgmap v971: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:54 smithi138 ceph-mon[33706]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T21:35:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:54 smithi088 ceph-mon[39202]: pgmap v971: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:54 smithi088 ceph-mon[39202]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T21:35:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:54 smithi088 ceph-mon[30075]: pgmap v971: 300 pgs: 300 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:54 smithi088 ceph-mon[30075]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T21:35:56.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:55 smithi138 ceph-mon[33706]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T21:35:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:55 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[39202]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T21:35:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[30075]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T21:35:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:56.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:56.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:56 smithi138 ceph-mon[33706]: pgmap v974: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:56 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:56 smithi138 ceph-mon[33706]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T21:35:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:56 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:56 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[39202]: pgmap v974: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[39202]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T21:35:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[30075]: pgmap v974: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[30075]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T21:35:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:35:57.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:56 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:58.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:57 smithi138 ceph-mon[33706]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T21:35:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:35:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:35:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[39202]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T21:35:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:35:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:35:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:35:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[30075]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T21:35:58.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:35:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:35:59.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:58 smithi138 ceph-mon[33706]: pgmap v977: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:58 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]': finished 2024-03-26T21:35:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:58 smithi138 ceph-mon[33706]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T21:35:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:58 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]: dispatch 2024-03-26T21:35:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:58 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]: dispatch 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[39202]: pgmap v977: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]': finished 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[39202]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]: dispatch 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]: dispatch 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[30075]: pgmap v977: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-128"}]': finished 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[30075]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T21:35:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]: dispatch 2024-03-26T21:35:59.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:58 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]: dispatch 2024-03-26T21:36:00.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:59 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:59 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]': finished 2024-03-26T21:36:00.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:35:59 smithi138 ceph-mon[33706]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T21:36:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:59 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:59 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]': finished 2024-03-26T21:36:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:35:59 smithi088 ceph-mon[39202]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T21:36:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:59 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:59 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-128", "mode": "writeback"}]': finished 2024-03-26T21:36:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:35:59 smithi088 ceph-mon[30075]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T21:36:01.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:00 smithi138 ceph-mon[33706]: pgmap v980: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:01.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:36:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:36:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:00 smithi088 ceph-mon[39202]: pgmap v980: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:00 smithi088 ceph-mon[30075]: pgmap v980: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:02.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:01 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:02.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:01 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:01 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:02.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:01 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:01 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:01 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:36:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:36:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:03.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:02 smithi138 ceph-mon[33706]: pgmap v981: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 356 B/s wr, 1 op/s 2024-03-26T21:36:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:02 smithi088 ceph-mon[30075]: pgmap v981: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 356 B/s wr, 1 op/s 2024-03-26T21:36:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:02 smithi088 ceph-mon[39202]: pgmap v981: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 356 B/s wr, 1 op/s 2024-03-26T21:36:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:04 smithi138 ceph-mon[33706]: pgmap v982: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 303 B/s wr, 1 op/s 2024-03-26T21:36:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:04 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:04 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:04 smithi088 ceph-mon[39202]: pgmap v982: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 303 B/s wr, 1 op/s 2024-03-26T21:36:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:04 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:04 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:04 smithi088 ceph-mon[30075]: pgmap v982: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 303 B/s wr, 1 op/s 2024-03-26T21:36:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:04 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:04 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:06.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:05 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:06.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:05 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:36:06.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:05 smithi138 ceph-mon[33706]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T21:36:06.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:05 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[39202]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[30075]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T21:36:06.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:05 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]: dispatch 2024-03-26T21:36:07.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:06 smithi138 ceph-mon[33706]: pgmap v984: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 264 B/s wr, 1 op/s 2024-03-26T21:36:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:06 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:06 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]': finished 2024-03-26T21:36:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:06 smithi138 ceph-mon[33706]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[39202]: pgmap v984: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 264 B/s wr, 1 op/s 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]': finished 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[39202]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[30075]: pgmap v984: 300 pgs: 300 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 264 B/s wr, 1 op/s 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-128"}]': finished 2024-03-26T21:36:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:06 smithi088 ceph-mon[30075]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T21:36:08.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:08 smithi138 ceph-mon[33706]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T21:36:08.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:08 smithi088 ceph-mon[39202]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T21:36:08.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:08 smithi088 ceph-mon[30075]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T21:36:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:09 smithi138 ceph-mon[33706]: pgmap v987: 268 pgs: 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:36:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:09 smithi138 ceph-mon[33706]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T21:36:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:09 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[39202]: pgmap v987: 268 pgs: 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[39202]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[30075]: pgmap v987: 268 pgs: 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[30075]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:09 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:10 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:10 smithi138 ceph-mon[33706]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T21:36:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:10.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:10 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[39202]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[30075]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:10.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:10 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:11.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[39202]: pgmap v990: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:11.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:11.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[39202]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[30075]: pgmap v990: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[30075]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:11.385 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:11.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:36:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:36:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:11 smithi138 ceph-mon[33706]: pgmap v990: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:11 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:11 smithi138 ceph-mon[33706]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T21:36:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:11 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]': finished 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]: dispatch 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]: dispatch 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]': finished 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[39202]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T21:36:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]': finished 2024-03-26T21:36:12.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T21:36:12.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]: dispatch 2024-03-26T21:36:12.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]: dispatch 2024-03-26T21:36:12.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:12.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:12.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]': finished 2024-03-26T21:36:12.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-mon[30075]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T21:36:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-130"}]': finished 2024-03-26T21:36:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T21:36:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]: dispatch 2024-03-26T21:36:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]: dispatch 2024-03-26T21:36:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-130", "mode": "writeback"}]': finished 2024-03-26T21:36:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:12 smithi138 ceph-mon[33706]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T21:36:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:36:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:36:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:13 smithi138 ceph-mon[33706]: pgmap v993: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:13 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:13 smithi088 ceph-mon[39202]: pgmap v993: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:13 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:13 smithi088 ceph-mon[30075]: pgmap v993: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:13 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:14.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:14 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:14 smithi138 ceph-mon[33706]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T21:36:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:14 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[39202]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T21:36:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[30075]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T21:36:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:14 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]: dispatch 2024-03-26T21:36:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:15 smithi138 ceph-mon[33706]: pgmap v996: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:15 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:15 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]': finished 2024-03-26T21:36:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:15 smithi138 ceph-mon[33706]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[39202]: pgmap v996: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]': finished 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[39202]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[30075]: pgmap v996: 300 pgs: 300 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-130"}]': finished 2024-03-26T21:36:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:15 smithi088 ceph-mon[30075]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T21:36:16.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:16 smithi138 ceph-mon[33706]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T21:36:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:16 smithi088 ceph-mon[30075]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T21:36:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:16 smithi088 ceph-mon[39202]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T21:36:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:17 smithi138 ceph-mon[33706]: pgmap v999: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:17 smithi138 ceph-mon[33706]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T21:36:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:17 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:17 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[39202]: pgmap v999: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[39202]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[30075]: pgmap v999: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[30075]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:17 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T21:36:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:36:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:36:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:36:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:18 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T21:36:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:36:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:36:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:18 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:19 smithi138 ceph-mon[33706]: pgmap v1002: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:19 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:19 smithi138 ceph-mon[33706]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T21:36:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:19 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[39202]: pgmap v1002: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[39202]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[30075]: pgmap v1002: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[30075]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:19 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:20.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:20 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:20 smithi138 ceph-mon[33706]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T21:36:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]: dispatch 2024-03-26T21:36:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:20 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]: dispatch 2024-03-26T21:36:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:20 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:20 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]': finished 2024-03-26T21:36:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[39202]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T21:36:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]: dispatch 2024-03-26T21:36:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]: dispatch 2024-03-26T21:36:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:20.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]': finished 2024-03-26T21:36:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[30075]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T21:36:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]: dispatch 2024-03-26T21:36:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]: dispatch 2024-03-26T21:36:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:20 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-132", "mode": "writeback"}]': finished 2024-03-26T21:36:21.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:36:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:36:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:21 smithi138 ceph-mon[33706]: pgmap v1005: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:21 smithi138 ceph-mon[33706]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T21:36:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:21 smithi088 ceph-mon[39202]: pgmap v1005: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:21 smithi088 ceph-mon[39202]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T21:36:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:21 smithi088 ceph-mon[30075]: pgmap v1005: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:21 smithi088 ceph-mon[30075]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T21:36:22.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:22 smithi088 ceph-mon[39202]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T21:36:22.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:22 smithi088 ceph-mon[30075]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T21:36:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:22 smithi138 ceph-mon[33706]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T21:36:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:36:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:36:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:23 smithi138 ceph-mon[33706]: pgmap v1008: 300 pgs: 300 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:23 smithi138 ceph-mon[33706]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T21:36:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:23 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[30075]: pgmap v1008: 300 pgs: 300 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[30075]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T21:36:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[39202]: pgmap v1008: 300 pgs: 300 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[39202]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T21:36:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:23 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:24.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:24 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:24 smithi138 ceph-mon[33706]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T21:36:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:24 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[39202]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T21:36:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[30075]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T21:36:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:24.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:24 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:25 smithi138 ceph-mon[33706]: pgmap v1011: 300 pgs: 300 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:25 smithi138 ceph-mon[33706]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T21:36:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:25 smithi088 ceph-mon[39202]: pgmap v1011: 300 pgs: 300 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:25 smithi088 ceph-mon[39202]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T21:36:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:25 smithi088 ceph-mon[30075]: pgmap v1011: 300 pgs: 300 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:36:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:25 smithi088 ceph-mon[30075]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T21:36:26.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:26 smithi138 ceph-mon[33706]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T21:36:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:26 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:26 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[30075]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[39202]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: pgmap v1014: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:36:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T21:36:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:27 smithi138 ceph-mon[33706]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: pgmap v1014: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[39202]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T21:36:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: pgmap v1014: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:36:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T21:36:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]: dispatch 2024-03-26T21:36:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-132"}]': finished 2024-03-26T21:36:27.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:27 smithi088 ceph-mon[30075]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T21:36:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:28 smithi138 ceph-mon[33706]: pgmap v1017: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:36:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:28 smithi138 ceph-mon[33706]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T21:36:29.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:28 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:28 smithi088 ceph-mon[39202]: pgmap v1017: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:36:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:28 smithi088 ceph-mon[39202]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T21:36:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:28 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:28 smithi088 ceph-mon[30075]: pgmap v1017: 300 pgs: 300 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:36:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:28 smithi088 ceph-mon[30075]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T21:36:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:28 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:30.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:29 smithi138 ceph-mon[33706]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T21:36:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:30.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:29 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:30.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:29 smithi088 ceph-mon[30075]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T21:36:30.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:30.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:29 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:29 smithi088 ceph-mon[39202]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T21:36:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:29 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:30 smithi138 ceph-mon[33706]: pgmap v1020: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:30 smithi138 ceph-mon[33706]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T21:36:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[39202]: pgmap v1020: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[39202]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T21:36:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[30075]: pgmap v1020: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[30075]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T21:36:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:31.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:31.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:36:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:36:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:31 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:31 smithi138 ceph-mon[33706]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T21:36:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:31 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:32.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:31 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[39202]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[30075]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T21:36:32.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:32.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:31 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:36:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:33.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:32 smithi138 ceph-mon[33706]: pgmap v1023: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:36:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:32 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]': finished 2024-03-26T21:36:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:32 smithi138 ceph-mon[33706]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T21:36:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:32 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]: dispatch 2024-03-26T21:36:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:32 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]: dispatch 2024-03-26T21:36:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[39202]: pgmap v1023: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:36:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]': finished 2024-03-26T21:36:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[39202]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T21:36:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]: dispatch 2024-03-26T21:36:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]: dispatch 2024-03-26T21:36:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[30075]: pgmap v1023: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:36:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-134"}]': finished 2024-03-26T21:36:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[30075]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T21:36:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]: dispatch 2024-03-26T21:36:33.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:32 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]: dispatch 2024-03-26T21:36:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:33 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]': finished 2024-03-26T21:36:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:33 smithi138 ceph-mon[33706]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T21:36:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:34.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]': finished 2024-03-26T21:36:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[39202]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T21:36:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:34.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-134", "mode": "writeback"}]': finished 2024-03-26T21:36:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[30075]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T21:36:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:34.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:34 smithi138 ceph-mon[33706]: pgmap v1026: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:34 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:34 smithi138 ceph-mon[33706]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T21:36:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:35.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:34 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[39202]: pgmap v1026: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[39202]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T21:36:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[30075]: pgmap v1026: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[30075]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T21:36:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:35.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:34 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]: dispatch 2024-03-26T21:36:36.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:35 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:35 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]': finished 2024-03-26T21:36:36.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:35 smithi138 ceph-mon[33706]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T21:36:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:35 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:35 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]': finished 2024-03-26T21:36:36.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:35 smithi088 ceph-mon[30075]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T21:36:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:35 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:35 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-134"}]': finished 2024-03-26T21:36:36.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:35 smithi088 ceph-mon[39202]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T21:36:37.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:36 smithi138 ceph-mon[33706]: pgmap v1029: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T21:36:37.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:36 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:36 smithi138 ceph-mon[33706]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T21:36:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:36 smithi088 ceph-mon[30075]: pgmap v1029: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T21:36:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:36 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:36 smithi088 ceph-mon[30075]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T21:36:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:36 smithi088 ceph-mon[39202]: pgmap v1029: 300 pgs: 300 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T21:36:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:36 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:36 smithi088 ceph-mon[39202]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T21:36:38.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:37 smithi138 ceph-mon[33706]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T21:36:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:38.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:37 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:37 smithi088 ceph-mon[30075]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T21:36:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:38.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:37 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:38.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:37 smithi088 ceph-mon[39202]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T21:36:38.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:38.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:37 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:38 smithi138 ceph-mon[33706]: pgmap v1032: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:36:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:38 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:38 smithi138 ceph-mon[33706]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T21:36:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:39.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:38 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[39202]: pgmap v1032: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:36:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[39202]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T21:36:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:39.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[30075]: pgmap v1032: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:36:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[30075]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T21:36:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:39.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:38 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:39 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:39 smithi138 ceph-mon[33706]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T21:36:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:40.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:39 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[39202]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[30075]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:40.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:39 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[39202]: pgmap v1035: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]': finished 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[39202]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]: dispatch 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]: dispatch 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[30075]: pgmap v1035: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]': finished 2024-03-26T21:36:41.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[30075]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T21:36:41.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]: dispatch 2024-03-26T21:36:41.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]: dispatch 2024-03-26T21:36:41.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:41.235 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:41.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:36:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:40 smithi138 ceph-mon[33706]: pgmap v1035: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:40 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-136"}]': finished 2024-03-26T21:36:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:40 smithi138 ceph-mon[33706]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T21:36:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]: dispatch 2024-03-26T21:36:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:40 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]: dispatch 2024-03-26T21:36:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:40 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:36:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]': finished 2024-03-26T21:36:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[30075]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]': finished 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[39202]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:42.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:41 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:36:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-136", "mode": "writeback"}]': finished 2024-03-26T21:36:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-mon[33706]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T21:36:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:42.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:41 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:36:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:36:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:43.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:43 smithi138 ceph-mon[33706]: pgmap v1038: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:43 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:43 smithi138 ceph-mon[33706]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T21:36:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:43 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:43 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[30075]: pgmap v1038: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[30075]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T21:36:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[39202]: pgmap v1038: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:36:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[39202]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T21:36:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:43 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]: dispatch 2024-03-26T21:36:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:44 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:44 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]': finished 2024-03-26T21:36:44.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:44 smithi138 ceph-mon[33706]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T21:36:44.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:44 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:44.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:44 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]': finished 2024-03-26T21:36:44.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:44 smithi088 ceph-mon[30075]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T21:36:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:44 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:36:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:44 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-136"}]': finished 2024-03-26T21:36:44.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:44 smithi088 ceph-mon[39202]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T21:36:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:45 smithi138 ceph-mon[33706]: pgmap v1041: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:45 smithi138 ceph-mon[33706]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T21:36:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:45 smithi088 ceph-mon[30075]: pgmap v1041: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:45 smithi088 ceph-mon[30075]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T21:36:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:45 smithi088 ceph-mon[39202]: pgmap v1041: 300 pgs: 300 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:45 smithi088 ceph-mon[39202]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T21:36:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:46 smithi138 ceph-mon[33706]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T21:36:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:46.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:46 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:46.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:46 smithi088 ceph-mon[30075]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T21:36:46.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:46.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:46 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:46.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:46 smithi088 ceph-mon[39202]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T21:36:46.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:46.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:46 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:47 smithi138 ceph-mon[33706]: pgmap v1044: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T21:36:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:47 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:47 smithi138 ceph-mon[33706]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T21:36:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:47 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:47 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[30075]: pgmap v1044: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[30075]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[39202]: pgmap v1044: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T21:36:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[39202]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T21:36:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:47 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:48.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:48 smithi138 ceph-mon[33706]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T21:36:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:36:48.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[39202]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[30075]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:36:48.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:36:49.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:49 smithi138 ceph-mon[33706]: pgmap v1047: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T21:36:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:49 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:36:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:49 smithi138 ceph-mon[33706]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T21:36:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:36:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:49 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:36:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[39202]: pgmap v1047: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T21:36:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:36:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[39202]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T21:36:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:36:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:36:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[30075]: pgmap v1047: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T21:36:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:36:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[30075]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T21:36:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:36:49.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:49 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:36:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:50 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:36:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:50 smithi138 ceph-mon[33706]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T21:36:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:36:50.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:50 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[39202]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[30075]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:36:50.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:50 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T21:36:51.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:36:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:36:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:51 smithi138 ceph-mon[33706]: pgmap v1050: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:51 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:36:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:51 smithi138 ceph-mon[33706]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T21:36:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:51 smithi088 ceph-mon[30075]: pgmap v1050: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:51 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:36:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:51 smithi088 ceph-mon[30075]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T21:36:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:51 smithi088 ceph-mon[39202]: pgmap v1050: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:36:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:51 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T21:36:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:51 smithi088 ceph-mon[39202]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T21:36:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]: dispatch 2024-03-26T21:36:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]: dispatch 2024-03-26T21:36:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:52.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:52.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]: dispatch 2024-03-26T21:36:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]: dispatch 2024-03-26T21:36:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:52.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:52.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:52.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:52 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:36:52.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]: dispatch 2024-03-26T21:36:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:52 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]: dispatch 2024-03-26T21:36:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:52.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:52 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:36:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:36:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:36:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:53 smithi138 ceph-mon[33706]: pgmap v1052: 300 pgs: 300 active+clean; 4.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:36:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:53 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]': finished 2024-03-26T21:36:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:53 smithi138 ceph-mon[33706]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T21:36:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:53 smithi088 ceph-mon[39202]: pgmap v1052: 300 pgs: 300 active+clean; 4.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:36:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:53 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]': finished 2024-03-26T21:36:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:53 smithi088 ceph-mon[39202]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T21:36:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:53 smithi088 ceph-mon[30075]: pgmap v1052: 300 pgs: 300 active+clean; 4.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:36:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:53 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-138"}]': finished 2024-03-26T21:36:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:53 smithi088 ceph-mon[30075]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T21:36:54.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:54 smithi138 ceph-mon[33706]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T21:36:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:54 smithi138 ceph-mon[33706]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T21:36:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:54.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:54 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[30075]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[30075]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[39202]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[39202]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:54.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:54 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:36:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:55 smithi138 ceph-mon[33706]: pgmap v1055: 268 pgs: 268 active+clean; 4.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:36:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:55 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:55 smithi138 ceph-mon[33706]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T21:36:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:55 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[39202]: pgmap v1055: 268 pgs: 268 active+clean; 4.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:36:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[39202]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T21:36:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[30075]: pgmap v1055: 268 pgs: 268 active+clean; 4.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:36:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:36:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[30075]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T21:36:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:55 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:36:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:56 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:56 smithi138 ceph-mon[33706]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T21:36:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:36:56.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:56 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[39202]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:36:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[30075]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T21:36:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:36:56.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:56 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T21:36:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:57 smithi138 ceph-mon[33706]: pgmap v1058: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T21:36:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:57 smithi138 ceph-mon[33706]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T21:36:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:36:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[30075]: pgmap v1058: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[30075]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[39202]: pgmap v1058: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[39202]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:36:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T21:36:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:59 smithi138 ceph-mon[33706]: pgmap v1061: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:59 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T21:36:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:59 smithi138 ceph-mon[33706]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T21:36:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:59 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:36:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:36:59 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[30075]: pgmap v1061: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[30075]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[39202]: pgmap v1061: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[39202]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:36:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:36:59 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:00.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:00 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:00 smithi138 ceph-mon[33706]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T21:37:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:00 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:37:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:00 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:37:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:00 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T21:37:00.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:00 smithi138 ceph-mon[33706]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[30075]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[30075]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[39202]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T21:37:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:37:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T21:37:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T21:37:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:00 smithi088 ceph-mon[39202]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T21:37:01.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:01 smithi138 ceph-mon[33706]: pgmap v1064: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:01.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:37:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:37:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:01 smithi088 ceph-mon[30075]: pgmap v1064: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:01 smithi088 ceph-mon[39202]: pgmap v1064: 300 pgs: 300 active+clean; 4.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:02.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:02 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:02.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:02 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:02.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:02 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:37:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:37:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:03 smithi138 ceph-mon[33706]: pgmap v1066: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-26T21:37:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:03 smithi088 ceph-mon[30075]: pgmap v1066: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-26T21:37:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:03 smithi088 ceph-mon[39202]: pgmap v1066: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-26T21:37:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:05 smithi138 ceph-mon[33706]: pgmap v1067: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 0 op/s 2024-03-26T21:37:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:05 smithi088 ceph-mon[39202]: pgmap v1067: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 0 op/s 2024-03-26T21:37:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:05 smithi088 ceph-mon[30075]: pgmap v1067: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 0 op/s 2024-03-26T21:37:07.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:07 smithi138 ceph-mon[33706]: pgmap v1068: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.4 KiB/s wr, 2 op/s 2024-03-26T21:37:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:07 smithi088 ceph-mon[30075]: pgmap v1068: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.4 KiB/s wr, 2 op/s 2024-03-26T21:37:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:07 smithi088 ceph-mon[39202]: pgmap v1068: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.4 KiB/s wr, 2 op/s 2024-03-26T21:37:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:09 smithi138 ceph-mon[33706]: pgmap v1069: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2024-03-26T21:37:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:09 smithi088 ceph-mon[30075]: pgmap v1069: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2024-03-26T21:37:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:09 smithi088 ceph-mon[39202]: pgmap v1069: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 KiB/s wr, 1 op/s 2024-03-26T21:37:11.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:37:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: pgmap v1070: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-26T21:37:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]: dispatch 2024-03-26T21:37:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]: dispatch 2024-03-26T21:37:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]: dispatch 2024-03-26T21:37:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]: dispatch 2024-03-26T21:37:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]: dispatch 2024-03-26T21:37:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]: dispatch 2024-03-26T21:37:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]: dispatch 2024-03-26T21:37:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]: dispatch 2024-03-26T21:37:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: pgmap v1070: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-26T21:37:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]: dispatch 2024-03-26T21:37:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]: dispatch 2024-03-26T21:37:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: pgmap v1070: 300 pgs: 300 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]: dispatch 2024-03-26T21:37:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]: dispatch 2024-03-26T21:37:11.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]': finished 2024-03-26T21:37:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]': finished 2024-03-26T21:37:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]': finished 2024-03-26T21:37:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]': finished 2024-03-26T21:37:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]': finished 2024-03-26T21:37:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]': finished 2024-03-26T21:37:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[39202]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]': finished 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]': finished 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]': finished 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]': finished 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]': finished 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]': finished 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-mon[30075]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T21:37:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.2", "id": [5, 0]}]': finished 2024-03-26T21:37:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [2, 0]}]': finished 2024-03-26T21:37:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.8", "id": [7, 3]}]': finished 2024-03-26T21:37:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.c", "id": [5, 0]}]': finished 2024-03-26T21:37:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.11", "id": [2, 3]}]': finished 2024-03-26T21:37:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3, 5, 4]}]': finished 2024-03-26T21:37:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T21:37:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:12.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:12 smithi138 ceph-mon[33706]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T21:37:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:37:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:37:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:13 smithi138 ceph-mon[33706]: pgmap v1072: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T21:37:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:13 smithi088 ceph-mon[39202]: pgmap v1072: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T21:37:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:13 smithi088 ceph-mon[30075]: pgmap v1072: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T21:37:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:15 smithi138 ceph-mon[33706]: pgmap v1074: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-03-26T21:37:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:15 smithi088 ceph-mon[39202]: pgmap v1074: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-03-26T21:37:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:15 smithi088 ceph-mon[30075]: pgmap v1074: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-03-26T21:37:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:16 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]: dispatch 2024-03-26T21:37:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:16 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]: dispatch 2024-03-26T21:37:16.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:16 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]: dispatch 2024-03-26T21:37:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:17 smithi138 ceph-mon[33706]: pgmap v1075: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s; 12 KiB/s, 0 objects/s recovering 2024-03-26T21:37:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:17 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]': finished 2024-03-26T21:37:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:17 smithi138 ceph-mon[33706]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T21:37:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:17 smithi088 ceph-mon[30075]: pgmap v1075: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s; 12 KiB/s, 0 objects/s recovering 2024-03-26T21:37:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:17 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]': finished 2024-03-26T21:37:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:17 smithi088 ceph-mon[30075]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T21:37:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:17 smithi088 ceph-mon[39202]: pgmap v1075: 300 pgs: 300 active+clean; 4.5 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s; 12 KiB/s, 0 objects/s recovering 2024-03-26T21:37:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:17 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-140"}]': finished 2024-03-26T21:37:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:17 smithi088 ceph-mon[39202]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T21:37:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:18 smithi138 ceph-mon[33706]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T21:37:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:18 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:37:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:18 smithi138 ceph-mon[33706]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T21:37:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:18 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[39202]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[39202]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[30075]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[30075]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T21:37:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:18 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: pgmap v1078: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:37:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:37:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:37:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:19 smithi138 ceph-mon[33706]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: pgmap v1078: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[30075]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: pgmap v1078: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:19 smithi088 ceph-mon[39202]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T21:37:20.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:20 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:20 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:20 smithi138 ceph-mon[33706]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T21:37:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:20 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:20 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[39202]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T21:37:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[30075]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T21:37:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:20.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:20 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:21.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:37:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:37:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:21 smithi138 ceph-mon[33706]: pgmap v1081: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:21 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]': finished 2024-03-26T21:37:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:21 smithi138 ceph-mon[33706]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T21:37:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]: dispatch 2024-03-26T21:37:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:21 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]: dispatch 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[39202]: pgmap v1081: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]': finished 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[39202]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]: dispatch 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]: dispatch 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[30075]: pgmap v1081: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-142"}]': finished 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[30075]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T21:37:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]: dispatch 2024-03-26T21:37:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:21 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]: dispatch 2024-03-26T21:37:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]': finished 2024-03-26T21:37:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[39202]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]': finished 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[30075]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:22.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:22 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:22 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-142", "mode": "writeback"}]': finished 2024-03-26T21:37:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:22 smithi138 ceph-mon[33706]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T21:37:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:22 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:37:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:37:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:23 smithi138 ceph-mon[33706]: pgmap v1084: 300 pgs: 300 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:37:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:23 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:37:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:23 smithi138 ceph-mon[33706]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T21:37:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:23 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:23 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[30075]: pgmap v1084: 300 pgs: 300 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:37:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:37:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[30075]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T21:37:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[39202]: pgmap v1084: 300 pgs: 300 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:37:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:37:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[39202]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T21:37:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:23 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:25 smithi138 ceph-mon[33706]: pgmap v1087: 300 pgs: 300 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:37:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:37:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:25 smithi138 ceph-mon[33706]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T21:37:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[39202]: pgmap v1087: 300 pgs: 300 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[39202]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[30075]: pgmap v1087: 300 pgs: 300 active+clean; 4.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[30075]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T21:37:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:37:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:26 smithi138 ceph-mon[33706]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[39202]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T21:37:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T21:37:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:37:26.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:26 smithi088 ceph-mon[30075]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T21:37:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:27 smithi138 ceph-mon[33706]: pgmap v1090: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:37:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:27 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:37:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:27 smithi088 ceph-mon[30075]: pgmap v1090: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:37:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:27 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:37:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:27 smithi088 ceph-mon[39202]: pgmap v1090: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:37:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:27 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T21:37:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:28 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T21:37:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:28 smithi138 ceph-mon[33706]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T21:37:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:37:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:28 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[39202]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[30075]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:37:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:28 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T21:37:29.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:29 smithi138 ceph-mon[33706]: pgmap v1093: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:29 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T21:37:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:29 smithi138 ceph-mon[33706]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T21:37:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:29.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:29 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[39202]: pgmap v1093: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[39202]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[30075]: pgmap v1093: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[30075]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:29 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:37:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:30 smithi138 ceph-mon[33706]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T21:37:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:30 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]': finished 2024-03-26T21:37:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:30 smithi138 ceph-mon[33706]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[39202]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]': finished 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[39202]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[30075]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T21:37:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]': finished 2024-03-26T21:37:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:30 smithi088 ceph-mon[30075]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T21:37:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[30075]: pgmap v1096: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[39202]: pgmap v1096: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:31 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:31.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:37:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:37:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:31 smithi138 ceph-mon[33706]: pgmap v1096: 300 pgs: 300 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:37:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:31 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:31 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:31 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-142"}]: dispatch 2024-03-26T21:37:32.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:32 smithi088 ceph-mon[30075]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T21:37:32.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:32 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:32 smithi088 ceph-mon[39202]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T21:37:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:32 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:32.484 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:37:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:37:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:32.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:32 smithi138 ceph-mon[33706]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T21:37:32.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:32 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:33 smithi138 ceph-mon[33706]: pgmap v1099: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:37:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:33 smithi138 ceph-mon[33706]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T21:37:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:33 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:33 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[39202]: pgmap v1099: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[39202]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[30075]: pgmap v1099: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[30075]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:33 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:34.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:34 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:34.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:34 smithi138 ceph-mon[33706]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T21:37:34.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:34 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:34.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:34 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:34.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:34 smithi138 ceph-mon[33706]: pgmap v1102: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[39202]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[39202]: pgmap v1102: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[30075]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:34.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:34 smithi088 ceph-mon[30075]: pgmap v1102: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:37:35.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:35 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:35 smithi138 ceph-mon[33706]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T21:37:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:35 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:35 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[39202]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[30075]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:35 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:36.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:36 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]': finished 2024-03-26T21:37:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:36 smithi138 ceph-mon[33706]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T21:37:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]: dispatch 2024-03-26T21:37:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:36 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]: dispatch 2024-03-26T21:37:36.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:36 smithi138 ceph-mon[33706]: pgmap v1105: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]': finished 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[39202]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]: dispatch 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]: dispatch 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[39202]: pgmap v1105: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-144"}]': finished 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[30075]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]: dispatch 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]: dispatch 2024-03-26T21:37:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:36 smithi088 ceph-mon[30075]: pgmap v1105: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:37:37.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:37 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:37 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]': finished 2024-03-26T21:37:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:37 smithi138 ceph-mon[33706]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T21:37:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:37 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:37 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]': finished 2024-03-26T21:37:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:37 smithi088 ceph-mon[39202]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T21:37:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:37 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:37 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-144", "mode": "readproxy"}]': finished 2024-03-26T21:37:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:37 smithi088 ceph-mon[30075]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T21:37:38.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:38 smithi138 ceph-mon[33706]: pgmap v1107: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 228 B/s wr, 1 op/s 2024-03-26T21:37:38.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:38 smithi088 ceph-mon[30075]: pgmap v1107: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 228 B/s wr, 1 op/s 2024-03-26T21:37:38.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:38 smithi088 ceph-mon[39202]: pgmap v1107: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 228 B/s wr, 1 op/s 2024-03-26T21:37:41.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:40 smithi138 ceph-mon[33706]: pgmap v1108: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T21:37:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:40 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:41.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:40 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:40 smithi088 ceph-mon[30075]: pgmap v1108: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T21:37:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:40 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:40 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:40 smithi088 ceph-mon[39202]: pgmap v1108: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T21:37:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:40 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:37:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:40 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:41.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:37:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:37:42.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:41 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:41 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:41 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:37:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:37:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:43.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:42 smithi138 ceph-mon[33706]: pgmap v1109: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 B/s wr, 1 op/s 2024-03-26T21:37:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:42 smithi088 ceph-mon[39202]: pgmap v1109: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 B/s wr, 1 op/s 2024-03-26T21:37:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:42 smithi088 ceph-mon[30075]: pgmap v1109: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 B/s wr, 1 op/s 2024-03-26T21:37:45.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:44 smithi138 ceph-mon[33706]: pgmap v1110: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T21:37:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:44 smithi088 ceph-mon[39202]: pgmap v1110: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T21:37:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:44 smithi088 ceph-mon[30075]: pgmap v1110: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T21:37:47.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:46 smithi138 ceph-mon[33706]: pgmap v1111: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:37:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:47.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:46 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:46 smithi088 ceph-mon[30075]: pgmap v1111: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:37:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:46 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:46 smithi088 ceph-mon[39202]: pgmap v1111: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:37:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:46 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:48.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:47 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:37:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:47 smithi138 ceph-mon[33706]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T21:37:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:48.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:47 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[30075]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[39202]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:47 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: pgmap v1113: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]': finished 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:49.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:48 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: pgmap v1113: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]': finished 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: pgmap v1113: 300 pgs: 300 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:37:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]': finished 2024-03-26T21:37:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T21:37:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:37:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:49.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:48 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-144"}]: dispatch 2024-03-26T21:37:50.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:49 smithi138 ceph-mon[33706]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T21:37:50.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:49 smithi088 ceph-mon[30075]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T21:37:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:49 smithi088 ceph-mon[39202]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T21:37:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:50 smithi138 ceph-mon[33706]: pgmap v1116: 268 pgs: 268 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:37:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:50 smithi138 ceph-mon[33706]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T21:37:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:51.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:50 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[30075]: pgmap v1116: 268 pgs: 268 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:37:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[30075]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T21:37:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[39202]: pgmap v1116: 268 pgs: 268 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:37:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[39202]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T21:37:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:51.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:50 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:37:51.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:37:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:37:52.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:51 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:51 smithi138 ceph-mon[33706]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T21:37:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:52.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:51 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[30075]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[39202]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:51 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:37:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:37:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:52 smithi138 ceph-mon[33706]: pgmap v1119: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:52 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:52 smithi138 ceph-mon[33706]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T21:37:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:37:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:52 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:37:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[30075]: pgmap v1119: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[30075]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T21:37:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:37:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:37:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[39202]: pgmap v1119: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:37:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[39202]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T21:37:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:37:53.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:52 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:37:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:53 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]': finished 2024-03-26T21:37:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:53 smithi138 ceph-mon[33706]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T21:37:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]: dispatch 2024-03-26T21:37:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:53 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]: dispatch 2024-03-26T21:37:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:53 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:53 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]': finished 2024-03-26T21:37:54.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:53 smithi138 ceph-mon[33706]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]': finished 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[39202]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]: dispatch 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]: dispatch 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]': finished 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[39202]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T21:37:54.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-79477-111", "overlaypool": "test-rados-api-smithi088-79477-146"}]': finished 2024-03-26T21:37:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[30075]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T21:37:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]: dispatch 2024-03-26T21:37:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]: dispatch 2024-03-26T21:37:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T21:37:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-79477-146", "mode": "writeback"}]': finished 2024-03-26T21:37:54.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:53 smithi088 ceph-mon[30075]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T21:37:55.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:54 smithi138 ceph-mon[33706]: pgmap v1122: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:55.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:55.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:54 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:54 smithi088 ceph-mon[30075]: pgmap v1122: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:54 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:54 smithi088 ceph-mon[39202]: pgmap v1122: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:37:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:54 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T21:37:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:55 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:37:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:55 smithi138 ceph-mon[33706]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T21:37:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:55 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:56.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:55 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[39202]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[30075]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T21:37:56.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:55 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:57.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:56 smithi138 ceph-mon[33706]: pgmap v1125: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:37:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:56 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:37:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:56 smithi138 ceph-mon[33706]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T21:37:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:56 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[39202]: pgmap v1125: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[39202]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[30075]: pgmap v1125: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[30075]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:56 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T21:37:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T21:37:58.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:37:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T21:37:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:37:58.225 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:57 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:37:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:37:58.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:57 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T21:37:59.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:58 smithi138 ceph-mon[33706]: pgmap v1128: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:37:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:58 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T21:37:59.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:58 smithi138 ceph-mon[33706]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T21:37:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:58 smithi088 ceph-mon[30075]: pgmap v1128: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:37:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:58 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T21:37:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:58 smithi088 ceph-mon[30075]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T21:37:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:58 smithi088 ceph-mon[39202]: pgmap v1128: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T21:37:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:58 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T21:37:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:58 smithi088 ceph-mon[39202]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T21:38:00.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:37:59 smithi138 ceph-mon[33706]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T21:38:00.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:37:59 smithi088 ceph-mon[30075]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T21:38:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:37:59 smithi088 ceph-mon[39202]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T21:38:01.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:00 smithi138 ceph-mon[33706]: pgmap v1131: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:00 smithi088 ceph-mon[39202]: pgmap v1131: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:00 smithi088 ceph-mon[30075]: pgmap v1131: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:01.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:38:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:38:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:38:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:38:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:03.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:02 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:02 smithi138 ceph-mon[33706]: pgmap v1132: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:02 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:02 smithi088 ceph-mon[39202]: pgmap v1132: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:02 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:02 smithi088 ceph-mon[30075]: pgmap v1132: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:05.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:04 smithi138 ceph-mon[33706]: pgmap v1133: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:04 smithi088 ceph-mon[39202]: pgmap v1133: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:04 smithi088 ceph-mon[30075]: pgmap v1133: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:07.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:06 smithi138 ceph-mon[33706]: pgmap v1134: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:06 smithi088 ceph-mon[39202]: pgmap v1134: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:06 smithi088 ceph-mon[30075]: pgmap v1134: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:09.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:08 smithi138 ceph-mon[33706]: pgmap v1135: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:08 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:09.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:08 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:08 smithi088 ceph-mon[39202]: pgmap v1135: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:08 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:08 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:08 smithi088 ceph-mon[30075]: pgmap v1135: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:08 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:08 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:10.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:09 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:10.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:09 smithi138 ceph-mon[33706]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T21:38:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:09 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:10.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:09 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[39202]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[30075]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:10.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:09 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:11.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: pgmap v1137: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]': finished 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:11.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:10 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: pgmap v1137: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]': finished 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: pgmap v1137: 300 pgs: 300 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]': finished 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-146"}]: dispatch 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:11.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:10 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:11.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:38:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:38:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:11 smithi088 ceph-mon[39202]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T21:38:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:11 smithi088 ceph-mon[30075]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T21:38:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:11 smithi138 ceph-mon[33706]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T21:38:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:38:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[39202]: pgmap v1140: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T21:38:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[39202]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T21:38:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[39202]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T21:38:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[30075]: pgmap v1140: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T21:38:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[30075]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T21:38:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[30075]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T21:38:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:13.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:12 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:12 smithi138 ceph-mon[33706]: pgmap v1140: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T21:38:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:12 smithi138 ceph-mon[33706]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T21:38:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:12 smithi138 ceph-mon[33706]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T21:38:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:12 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:13 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:13 smithi138 ceph-mon[33706]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T21:38:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:38:14.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:13 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:38:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[39202]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T21:38:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:38:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:38:14.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[30075]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T21:38:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:38:14.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:13 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T21:38:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:14 smithi138 ceph-mon[33706]: pgmap v1143: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:38:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:14 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:38:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:14 smithi138 ceph-mon[33706]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T21:38:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:14 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[39202]: pgmap v1143: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[39202]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[30075]: pgmap v1143: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[30075]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T21:38:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:15.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:14 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:16 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]': finished 2024-03-26T21:38:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:16 smithi138 ceph-mon[33706]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T21:38:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:16 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:16 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:16.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:16 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]': finished 2024-03-26T21:38:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[39202]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T21:38:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]': finished 2024-03-26T21:38:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[30075]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T21:38:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:16.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:16 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-148"}]: dispatch 2024-03-26T21:38:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:17 smithi138 ceph-mon[33706]: pgmap v1146: 300 pgs: 300 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:38:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:17 smithi138 ceph-mon[33706]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:17.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:17 smithi138 ceph-mon[33706]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T21:38:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:17 smithi088 ceph-mon[39202]: pgmap v1146: 300 pgs: 300 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:38:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:17 smithi088 ceph-mon[39202]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:17 smithi088 ceph-mon[39202]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T21:38:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:17 smithi088 ceph-mon[30075]: pgmap v1146: 300 pgs: 300 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T21:38:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:17 smithi088 ceph-mon[30075]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:17 smithi088 ceph-mon[30075]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T21:38:18.296 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:18 smithi138 ceph-mon[33706]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T21:38:18.296 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:18 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:18.296 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:18 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:18 smithi088 ceph-mon[39202]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T21:38:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:18 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:18 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:18.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:18 smithi088 ceph-mon[30075]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T21:38:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:18 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:18.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:18 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:18.723 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:38:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=sqlstore.transactions t=2024-03-26T21:38:18.304215724Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T21:38:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:19 smithi138 ceph-mon[33706]: pgmap v1149: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:38:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:19 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:19 smithi138 ceph-mon[33706]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T21:38:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:19 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:19 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-150"}]: dispatch 2024-03-26T21:38:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:19 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-150"}]: dispatch 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[30075]: pgmap v1149: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[30075]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-150"}]: dispatch 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-150"}]: dispatch 2024-03-26T21:38:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[39202]: pgmap v1149: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T21:38:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[39202]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T21:38:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-150"}]: dispatch 2024-03-26T21:38:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:19 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-150"}]: dispatch 2024-03-26T21:38:20.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:20 smithi138 ceph-mon[33706]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T21:38:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:20 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:38:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:20 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:20.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:20 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[39202]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[30075]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:20.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:20 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:21.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:38:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-mon[33706]: pgmap v1152: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-mon[33706]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T21:38:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:38:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:38:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:21 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[39202]: pgmap v1152: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[39202]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[30075]: pgmap v1152: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[30075]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T21:38:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:38:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:38:21.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:21 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[30075]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T21:38:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:22.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-152"}]: dispatch 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-152"}]: dispatch 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[30075]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[39202]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-152"}]: dispatch 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-152"}]: dispatch 2024-03-26T21:38:22.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-mon[39202]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:22 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:22 smithi138 ceph-mon[33706]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T21:38:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:22.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:22 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:22 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-152"}]: dispatch 2024-03-26T21:38:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:22 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-152"}]: dispatch 2024-03-26T21:38:22.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:22 smithi138 ceph-mon[33706]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:38:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:38:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:23 smithi138 ceph-mon[33706]: pgmap v1155: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:38:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:23 smithi138 ceph-mon[33706]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T21:38:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:23 smithi088 ceph-mon[30075]: pgmap v1155: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:38:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:23 smithi088 ceph-mon[30075]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T21:38:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:23 smithi088 ceph-mon[39202]: pgmap v1155: 300 pgs: 32 creating+peering, 268 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:38:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:23 smithi088 ceph-mon[39202]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T21:38:24.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:24 smithi138 ceph-mon[33706]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T21:38:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:24 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:24 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:24 smithi088 ceph-mon[39202]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T21:38:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:24 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:24 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:24 smithi088 ceph-mon[30075]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T21:38:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:24 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:24 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:38:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: pgmap v1158: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-111","var": "dedup_tier","val": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-111","var": "dedup_tier","val": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:25 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: pgmap v1158: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:38:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T21:38:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-111","var": "dedup_tier","val": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-111","var": "dedup_tier","val": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: pgmap v1158: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-79477-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-111","var": "dedup_tier","val": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi088-79477-111","var": "dedup_tier","val": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:25.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:25 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi088-79477-111", "tierpool": "test-rados-api-smithi088-79477-154"}]: dispatch 2024-03-26T21:38:26.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:26 smithi138 ceph-mon[33706]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T21:38:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:26 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:26 smithi088 ceph-mon[30075]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T21:38:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:26 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:26 smithi088 ceph-mon[39202]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T21:38:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:26 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:38:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:27 smithi138 ceph-mon[33706]: pgmap v1161: 268 pgs: 268 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T21:38:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:27 smithi138 ceph-mon[33706]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T21:38:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:27 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:27 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[39202]: pgmap v1161: 268 pgs: 268 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[39202]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[30075]: pgmap v1161: 268 pgs: 268 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[30075]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:27 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:28.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7905 ms) 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13080 ms) 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8061 ms) 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12838 ms) 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8052 ms) 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8061 ms) 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: ok, hit_set contains 320:602f83fe:::foo:head 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9055 ms) 2024-03-26T21:38:28.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first is 1711489021 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,1711489029,1711489030,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,1711489029,1711489030,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,1711489029,1711489030,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,1711489029,1711489033,0 2024-03-26T21:38:28.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,1711489029,1711489033,0 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489021,1711489023,1711489024,1711489026,1711489027,1711489029,1711489033,0 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1711489024,1711489026,1711489027,1711489029,1711489033,1711489035,1711489036,0 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first now 1711489024, trimmed 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (24087 ms) 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: foo0 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14071 ms) 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17587 ms) 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22155 ms) 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5040 ms) 2024-03-26T21:38:28.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3019 ms) 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3030 ms) 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3021 ms) 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (234506 ms total) 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-03-26T21:38:28.103 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-26T21:38:28.104 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (845905 ms total) 2024-03-26T21:38:28.104 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-26T21:38:28.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79190 2024-03-26T21:38:28.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79190 2024-03-26T21:38:28.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79322 2024-03-26T21:38:28.106 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79322 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79954 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79954 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79712 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79712 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79504 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79504 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79265 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79265 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79562 2024-03-26T21:38:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79562 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80095 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80095 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80146 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80146 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79278 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79278 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79989 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79989 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79875 2024-03-26T21:38:28.108 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79875 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79163 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79163 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79224 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79224 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79895 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79895 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79154 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79154 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79175 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79175 2024-03-26T21:38:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80405 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80405 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79339 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79339 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80477 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80477 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79527 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79527 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80051 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80051 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:38:28.110 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80556 2024-03-26T21:38:28.111 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80556 2024-03-26T21:38:28.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:28 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:28 smithi138 ceph-mon[33706]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T21:38:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:28.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:28 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[30075]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T21:38:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:28.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:28.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[39202]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T21:38:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4154974085' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:28 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]: dispatch 2024-03-26T21:38:29.374 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:29 smithi138 ceph-mon[33706]: pgmap v1164: 260 pgs: 260 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:29.374 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:29 smithi138 ceph-mon[33706]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:29.374 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:29 smithi138 ceph-mon[33706]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:29.374 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:29 smithi138 ceph-mon[33706]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T21:38:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[30075]: pgmap v1164: 260 pgs: 260 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[30075]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[30075]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[30075]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T21:38:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[39202]: pgmap v1164: 260 pgs: 260 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[39202]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:38:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[39202]: from='client.57161 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi088-79477-111"}]': finished 2024-03-26T21:38:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:29 smithi088 ceph-mon[39202]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T21:38:31.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:38:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:38:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:31 smithi138 ceph-mon[33706]: pgmap v1166: 260 pgs: 260 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:31 smithi088 ceph-mon[30075]: pgmap v1166: 260 pgs: 260 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:31 smithi088 ceph-mon[39202]: pgmap v1166: 260 pgs: 260 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:38:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:38:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:33 smithi138 ceph-mon[33706]: pgmap v1167: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:33 smithi088 ceph-mon[30075]: pgmap v1167: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:33 smithi088 ceph-mon[39202]: pgmap v1167: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:38:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:35 smithi138 ceph-mon[33706]: pgmap v1168: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-03-26T21:38:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:35 smithi088 ceph-mon[39202]: pgmap v1168: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-03-26T21:38:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:35 smithi088 ceph-mon[30075]: pgmap v1168: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-03-26T21:38:37.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:37 smithi138 ceph-mon[33706]: pgmap v1169: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:37 smithi088 ceph-mon[39202]: pgmap v1169: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:37 smithi088 ceph-mon[30075]: pgmap v1169: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:39 smithi138 ceph-mon[33706]: pgmap v1170: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:38:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:39 smithi088 ceph-mon[39202]: pgmap v1170: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:38:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:39 smithi088 ceph-mon[30075]: pgmap v1170: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:38:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:41 smithi138 ceph-mon[33706]: pgmap v1171: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T21:38:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:41.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:38:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:38:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:41 smithi088 ceph-mon[39202]: pgmap v1171: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T21:38:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:41 smithi088 ceph-mon[30075]: pgmap v1171: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T21:38:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:38:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:38:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:43.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:43 smithi138 ceph-mon[33706]: pgmap v1172: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:43 smithi088 ceph-mon[30075]: pgmap v1172: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:43 smithi088 ceph-mon[39202]: pgmap v1172: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:45.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:45 smithi138 ceph-mon[33706]: pgmap v1173: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:45 smithi088 ceph-mon[39202]: pgmap v1173: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:45 smithi088 ceph-mon[30075]: pgmap v1173: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:47.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:47 smithi138 ceph-mon[33706]: pgmap v1174: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:47 smithi088 ceph-mon[30075]: pgmap v1174: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:47 smithi088 ceph-mon[39202]: pgmap v1174: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:49.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:49 smithi138 ceph-mon[33706]: pgmap v1175: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:49 smithi088 ceph-mon[30075]: pgmap v1175: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:49 smithi088 ceph-mon[39202]: pgmap v1175: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:51 smithi138 ceph-mon[33706]: pgmap v1176: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:38:51.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:38:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:38:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:51 smithi088 ceph-mon[30075]: pgmap v1176: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:38:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:51 smithi088 ceph-mon[39202]: pgmap v1176: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:38:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:38:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:38:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:53 smithi138 ceph-mon[33706]: pgmap v1177: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:53 smithi088 ceph-mon[30075]: pgmap v1177: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:53 smithi088 ceph-mon[39202]: pgmap v1177: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:55 smithi138 ceph-mon[33706]: pgmap v1178: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:55 smithi088 ceph-mon[30075]: pgmap v1178: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:55 smithi088 ceph-mon[39202]: pgmap v1178: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:56.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:57 smithi138 ceph-mon[33706]: pgmap v1179: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:57 smithi088 ceph-mon[30075]: pgmap v1179: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:57 smithi088 ceph-mon[39202]: pgmap v1179: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:38:59.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:38:59 smithi138 ceph-mon[33706]: pgmap v1180: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:38:59 smithi088 ceph-mon[30075]: pgmap v1180: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:38:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:38:59 smithi088 ceph-mon[39202]: pgmap v1180: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:01.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:01 smithi138 ceph-mon[33706]: pgmap v1181: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:01.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:39:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:39:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:01 smithi088 ceph-mon[30075]: pgmap v1181: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:01 smithi088 ceph-mon[39202]: pgmap v1181: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:39:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:39:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:03 smithi138 ceph-mon[33706]: pgmap v1182: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:03 smithi088 ceph-mon[30075]: pgmap v1182: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:03 smithi088 ceph-mon[39202]: pgmap v1182: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:05 smithi138 ceph-mon[33706]: pgmap v1183: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:05 smithi088 ceph-mon[39202]: pgmap v1183: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:05 smithi088 ceph-mon[30075]: pgmap v1183: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:07 smithi138 ceph-mon[33706]: pgmap v1184: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:07 smithi088 ceph-mon[39202]: pgmap v1184: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:07 smithi088 ceph-mon[30075]: pgmap v1184: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:09 smithi138 ceph-mon[33706]: pgmap v1185: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:09 smithi088 ceph-mon[39202]: pgmap v1185: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:09 smithi088 ceph-mon[30075]: pgmap v1185: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:11.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:11 smithi138 ceph-mon[33706]: pgmap v1186: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:11.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:39:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:39:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:11 smithi088 ceph-mon[39202]: pgmap v1186: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:11 smithi088 ceph-mon[30075]: pgmap v1186: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:39:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:39:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:13 smithi138 ceph-mon[33706]: pgmap v1187: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:13 smithi088 ceph-mon[39202]: pgmap v1187: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:13 smithi088 ceph-mon[30075]: pgmap v1187: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:15 smithi138 ceph-mon[33706]: pgmap v1188: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:15 smithi088 ceph-mon[39202]: pgmap v1188: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:15 smithi088 ceph-mon[30075]: pgmap v1188: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:17 smithi138 ceph-mon[33706]: pgmap v1189: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:17 smithi088 ceph-mon[39202]: pgmap v1189: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:17 smithi088 ceph-mon[30075]: pgmap v1189: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:19 smithi138 ceph-mon[33706]: pgmap v1190: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:19 smithi088 ceph-mon[30075]: pgmap v1190: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:19 smithi088 ceph-mon[39202]: pgmap v1190: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:21.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:39:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:39:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:21 smithi138 ceph-mon[33706]: pgmap v1191: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:39:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:39:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:39:21.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:21 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[30075]: pgmap v1191: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[39202]: pgmap v1191: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:39:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:21 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:39:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:39:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:39:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:23 smithi138 ceph-mon[33706]: pgmap v1192: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:23 smithi088 ceph-mon[30075]: pgmap v1192: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:23 smithi088 ceph-mon[39202]: pgmap v1192: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:25 smithi138 ceph-mon[33706]: pgmap v1193: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:25 smithi088 ceph-mon[30075]: pgmap v1193: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:25 smithi088 ceph-mon[39202]: pgmap v1193: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:26.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:27 smithi138 ceph-mon[33706]: pgmap v1194: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:27 smithi088 ceph-mon[30075]: pgmap v1194: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:27 smithi088 ceph-mon[39202]: pgmap v1194: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:29.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:29 smithi138 ceph-mon[33706]: pgmap v1195: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:29 smithi088 ceph-mon[30075]: pgmap v1195: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:29 smithi088 ceph-mon[39202]: pgmap v1195: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:31.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:39:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:39:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:31 smithi138 ceph-mon[33706]: pgmap v1196: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:31 smithi088 ceph-mon[39202]: pgmap v1196: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:31 smithi088 ceph-mon[30075]: pgmap v1196: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:39:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:39:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:33 smithi138 ceph-mon[33706]: pgmap v1197: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:33 smithi088 ceph-mon[30075]: pgmap v1197: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:33 smithi088 ceph-mon[39202]: pgmap v1197: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:35 smithi138 ceph-mon[33706]: pgmap v1198: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:35 smithi088 ceph-mon[30075]: pgmap v1198: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:35 smithi088 ceph-mon[39202]: pgmap v1198: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:37.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:37 smithi138 ceph-mon[33706]: pgmap v1199: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:37 smithi088 ceph-mon[39202]: pgmap v1199: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:37 smithi088 ceph-mon[30075]: pgmap v1199: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:39 smithi138 ceph-mon[33706]: pgmap v1200: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:39 smithi088 ceph-mon[39202]: pgmap v1200: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:39 smithi088 ceph-mon[30075]: pgmap v1200: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:41 smithi138 ceph-mon[33706]: pgmap v1201: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:41.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:39:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:39:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:41 smithi088 ceph-mon[39202]: pgmap v1201: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:41 smithi088 ceph-mon[30075]: pgmap v1201: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:39:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:39:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:43.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:43 smithi138 ceph-mon[33706]: pgmap v1202: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:43 smithi088 ceph-mon[39202]: pgmap v1202: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:43 smithi088 ceph-mon[30075]: pgmap v1202: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:45.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:45 smithi138 ceph-mon[33706]: pgmap v1203: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:45 smithi088 ceph-mon[39202]: pgmap v1203: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:45 smithi088 ceph-mon[30075]: pgmap v1203: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:47 smithi088 ceph-mon[39202]: pgmap v1204: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:47 smithi088 ceph-mon[30075]: pgmap v1204: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:47.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:47 smithi138 ceph-mon[33706]: pgmap v1204: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:49 smithi088 ceph-mon[30075]: pgmap v1205: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:49 smithi088 ceph-mon[39202]: pgmap v1205: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:49.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:49 smithi138 ceph-mon[33706]: pgmap v1205: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:51 smithi088 ceph-mon[30075]: pgmap v1206: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:51 smithi088 ceph-mon[39202]: pgmap v1206: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:51.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:51 smithi138 ceph-mon[33706]: pgmap v1206: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:51.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:39:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:39:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:39:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:39:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:53.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:53 smithi138 ceph-mon[33706]: pgmap v1207: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:53 smithi088 ceph-mon[30075]: pgmap v1207: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:53 smithi088 ceph-mon[39202]: pgmap v1207: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:55.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:55 smithi138 ceph-mon[33706]: pgmap v1208: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:55.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:55 smithi088 ceph-mon[39202]: pgmap v1208: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:55.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:55 smithi088 ceph-mon[30075]: pgmap v1208: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:56.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:56.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:56.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:57.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:57 smithi138 ceph-mon[33706]: pgmap v1209: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:57 smithi088 ceph-mon[39202]: pgmap v1209: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:57.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:57 smithi088 ceph-mon[30075]: pgmap v1209: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:59.526 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:39:59 smithi138 ceph-mon[33706]: pgmap v1210: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:59.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:39:59 smithi088 ceph-mon[39202]: pgmap v1210: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:39:59.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:39:59 smithi088 ceph-mon[30075]: pgmap v1210: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:00.642 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:00 smithi138 ceph-mon[33706]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:40:00.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:00 smithi088 ceph-mon[39202]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:40:00.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:00 smithi088 ceph-mon[30075]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:40:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:01 smithi138 ceph-mon[33706]: pgmap v1211: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:01.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:40:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:40:01.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:01 smithi088 ceph-mon[39202]: pgmap v1211: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:01 smithi088 ceph-mon[30075]: pgmap v1211: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:40:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:40:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:03.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:03 smithi138 ceph-mon[33706]: pgmap v1212: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:03 smithi088 ceph-mon[39202]: pgmap v1212: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:03 smithi088 ceph-mon[30075]: pgmap v1212: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:05.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:05 smithi138 ceph-mon[33706]: pgmap v1213: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:05 smithi088 ceph-mon[30075]: pgmap v1213: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:05 smithi088 ceph-mon[39202]: pgmap v1213: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:07.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:07 smithi138 ceph-mon[33706]: pgmap v1214: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:07 smithi088 ceph-mon[30075]: pgmap v1214: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:07 smithi088 ceph-mon[39202]: pgmap v1214: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:08.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:08 smithi138 ceph-mon[33706]: pgmap v1215: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:08.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:08 smithi088 ceph-mon[39202]: pgmap v1215: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:08.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:08 smithi088 ceph-mon[30075]: pgmap v1215: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:11.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:10 smithi138 ceph-mon[33706]: pgmap v1216: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:10 smithi088 ceph-mon[30075]: pgmap v1216: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:10 smithi088 ceph-mon[39202]: pgmap v1216: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:11.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:40:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:40:12.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:12.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:40:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:40:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:13.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:13.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:12 smithi138 ceph-mon[33706]: pgmap v1217: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:12 smithi088 ceph-mon[30075]: pgmap v1217: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:12 smithi088 ceph-mon[39202]: pgmap v1217: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:15.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:14 smithi138 ceph-mon[33706]: pgmap v1218: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:14 smithi088 ceph-mon[30075]: pgmap v1218: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:14 smithi088 ceph-mon[39202]: pgmap v1218: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:17.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:16 smithi138 ceph-mon[33706]: pgmap v1219: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:16 smithi088 ceph-mon[39202]: pgmap v1219: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:16 smithi088 ceph-mon[30075]: pgmap v1219: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:19.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:18 smithi138 ceph-mon[33706]: pgmap v1220: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:18 smithi088 ceph-mon[30075]: pgmap v1220: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:18 smithi088 ceph-mon[39202]: pgmap v1220: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:21.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:20 smithi138 ceph-mon[33706]: pgmap v1221: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:21.232 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:20 smithi088 ceph-mon[30075]: pgmap v1221: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:20 smithi088 ceph-mon[39202]: pgmap v1221: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:21.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:40:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:40:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:40:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:40:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:40:22.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:21 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:40:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:40:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:40:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:40:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:40:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:40:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:40:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:40:22.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:21 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:40:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:40:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:40:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:23.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:22 smithi138 ceph-mon[33706]: pgmap v1222: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:22 smithi088 ceph-mon[30075]: pgmap v1222: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:22 smithi088 ceph-mon[39202]: pgmap v1222: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:25.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:24 smithi138 ceph-mon[33706]: pgmap v1223: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:25.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:24 smithi088 ceph-mon[39202]: pgmap v1223: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:25.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:24 smithi088 ceph-mon[30075]: pgmap v1223: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:27.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:26 smithi138 ceph-mon[33706]: pgmap v1224: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:26 smithi088 ceph-mon[30075]: pgmap v1224: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:26 smithi088 ceph-mon[39202]: pgmap v1224: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:29.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:28 smithi138 ceph-mon[33706]: pgmap v1225: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:28 smithi088 ceph-mon[39202]: pgmap v1225: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:28 smithi088 ceph-mon[30075]: pgmap v1225: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:31.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:30 smithi138 ceph-mon[33706]: pgmap v1226: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:30 smithi088 ceph-mon[30075]: pgmap v1226: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:30 smithi088 ceph-mon[39202]: pgmap v1226: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:31.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:40:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:40:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:40:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:40:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:33.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:32 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:32 smithi138 ceph-mon[33706]: pgmap v1227: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:33.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:32 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:32 smithi088 ceph-mon[39202]: pgmap v1227: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:32 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:32 smithi088 ceph-mon[30075]: pgmap v1227: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:35.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:34 smithi138 ceph-mon[33706]: pgmap v1228: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:35.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:34 smithi088 ceph-mon[39202]: pgmap v1228: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:34 smithi088 ceph-mon[30075]: pgmap v1228: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:37.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:36 smithi138 ceph-mon[33706]: pgmap v1229: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:36 smithi088 ceph-mon[39202]: pgmap v1229: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:36 smithi088 ceph-mon[30075]: pgmap v1229: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:39.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:38 smithi138 ceph-mon[33706]: pgmap v1230: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:38 smithi088 ceph-mon[39202]: pgmap v1230: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:39.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:38 smithi088 ceph-mon[30075]: pgmap v1230: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:41.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:40 smithi138 ceph-mon[33706]: pgmap v1231: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:40 smithi088 ceph-mon[39202]: pgmap v1231: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:40 smithi088 ceph-mon[30075]: pgmap v1231: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:41.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:40:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:40:42.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:40:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:40:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:43.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:42 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:43.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:42 smithi138 ceph-mon[33706]: pgmap v1232: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:42 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:42 smithi088 ceph-mon[30075]: pgmap v1232: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:42 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:42 smithi088 ceph-mon[39202]: pgmap v1232: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:45.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:44 smithi138 ceph-mon[33706]: pgmap v1233: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:44 smithi088 ceph-mon[30075]: pgmap v1233: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:44 smithi088 ceph-mon[39202]: pgmap v1233: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:47.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:46 smithi138 ceph-mon[33706]: pgmap v1234: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:47.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:46 smithi088 ceph-mon[30075]: pgmap v1234: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:46 smithi088 ceph-mon[39202]: pgmap v1234: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:49.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:48 smithi138 ceph-mon[33706]: pgmap v1235: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:49.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:48 smithi088 ceph-mon[30075]: pgmap v1235: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:48 smithi088 ceph-mon[39202]: pgmap v1235: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:50 smithi138 ceph-mon[33706]: pgmap v1236: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:51.232 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:50 smithi088 ceph-mon[30075]: pgmap v1236: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:50 smithi088 ceph-mon[39202]: pgmap v1236: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:40:51.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:40:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:40:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:40:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:40:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:53.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:53.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:52 smithi138 ceph-mon[33706]: pgmap v1237: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:52 smithi088 ceph-mon[30075]: pgmap v1237: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:52 smithi088 ceph-mon[39202]: pgmap v1237: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:55.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:54 smithi138 ceph-mon[33706]: pgmap v1238: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:54 smithi088 ceph-mon[39202]: pgmap v1238: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:54 smithi088 ceph-mon[30075]: pgmap v1238: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:57.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:56 smithi138 ceph-mon[33706]: pgmap v1239: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:56 smithi088 ceph-mon[30075]: pgmap v1239: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:56 smithi088 ceph-mon[39202]: pgmap v1239: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:40:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:59.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:40:58 smithi138 ceph-mon[33706]: pgmap v1240: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:40:58 smithi088 ceph-mon[30075]: pgmap v1240: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:40:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:40:58 smithi088 ceph-mon[39202]: pgmap v1240: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:01.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:00 smithi138 ceph-mon[33706]: pgmap v1241: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:00 smithi088 ceph-mon[30075]: pgmap v1241: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:00 smithi088 ceph-mon[39202]: pgmap v1241: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:01.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:41:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:41:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:41:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:41:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:02 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:03.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:02 smithi138 ceph-mon[33706]: pgmap v1242: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:02 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:02 smithi088 ceph-mon[30075]: pgmap v1242: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:02 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:02 smithi088 ceph-mon[39202]: pgmap v1242: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:05.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:04 smithi138 ceph-mon[33706]: pgmap v1243: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:04 smithi088 ceph-mon[39202]: pgmap v1243: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:05.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:04 smithi088 ceph-mon[30075]: pgmap v1243: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:07.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:06 smithi138 ceph-mon[33706]: pgmap v1244: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:06 smithi088 ceph-mon[39202]: pgmap v1244: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:07.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:06 smithi088 ceph-mon[30075]: pgmap v1244: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:09.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:08 smithi138 ceph-mon[33706]: pgmap v1245: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:09.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:08 smithi088 ceph-mon[30075]: pgmap v1245: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:09.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:08 smithi088 ceph-mon[39202]: pgmap v1245: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:11.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:10 smithi138 ceph-mon[33706]: pgmap v1246: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:11.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:10 smithi088 ceph-mon[30075]: pgmap v1246: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:10 smithi088 ceph-mon[39202]: pgmap v1246: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:11.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:41:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:41:12.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:12.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:41:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:41:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:13.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:13.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:12 smithi138 ceph-mon[33706]: pgmap v1247: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:13.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:12 smithi088 ceph-mon[30075]: pgmap v1247: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:12 smithi088 ceph-mon[39202]: pgmap v1247: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:15.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:14 smithi138 ceph-mon[33706]: pgmap v1248: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:15.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:14 smithi088 ceph-mon[39202]: pgmap v1248: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:15.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:14 smithi088 ceph-mon[30075]: pgmap v1248: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:17.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:16 smithi138 ceph-mon[33706]: pgmap v1249: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:17.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:16 smithi088 ceph-mon[39202]: pgmap v1249: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:17.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:16 smithi088 ceph-mon[30075]: pgmap v1249: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:19.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:18 smithi138 ceph-mon[33706]: pgmap v1250: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:19.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:18 smithi088 ceph-mon[39202]: pgmap v1250: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:19.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:18 smithi088 ceph-mon[30075]: pgmap v1250: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:21.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:20 smithi138 ceph-mon[33706]: pgmap v1251: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:20 smithi088 ceph-mon[39202]: pgmap v1251: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:21.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:20 smithi088 ceph-mon[30075]: pgmap v1251: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:21.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:41:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:41:22.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:21 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:41:22.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:21 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:41:22.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:21 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:41:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:41:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:23.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:22 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:23.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:22 smithi138 ceph-mon[33706]: pgmap v1252: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:23.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:22 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:41:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:22 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:41:23.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:22 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[30075]: pgmap v1252: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[39202]: pgmap v1252: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:41:23.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:22 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:41:25.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:24 smithi138 ceph-mon[33706]: pgmap v1253: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:25.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:24 smithi088 ceph-mon[39202]: pgmap v1253: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:25.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:24 smithi088 ceph-mon[30075]: pgmap v1253: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:27.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:26 smithi138 ceph-mon[33706]: pgmap v1254: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:27.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:26 smithi088 ceph-mon[39202]: pgmap v1254: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:27.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:26 smithi088 ceph-mon[30075]: pgmap v1254: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:27.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:29.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:28 smithi138 ceph-mon[33706]: pgmap v1255: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:28 smithi088 ceph-mon[39202]: pgmap v1255: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:29.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:28 smithi088 ceph-mon[30075]: pgmap v1255: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:31.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:30 smithi138 ceph-mon[33706]: pgmap v1256: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:31.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:30 smithi088 ceph-mon[39202]: pgmap v1256: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:31.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:30 smithi088 ceph-mon[30075]: pgmap v1256: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:31.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:41:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:41:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:41:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:41:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:32 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:33.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:32 smithi138 ceph-mon[33706]: pgmap v1257: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:32 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:32 smithi088 ceph-mon[39202]: pgmap v1257: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:32 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:33.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:32 smithi088 ceph-mon[30075]: pgmap v1257: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:35.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:34 smithi138 ceph-mon[33706]: pgmap v1258: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:35.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:34 smithi088 ceph-mon[39202]: pgmap v1258: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:35.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:34 smithi088 ceph-mon[30075]: pgmap v1258: 260 pgs: 260 active+clean; 582 KiB data, 716 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:37.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:36 smithi138 ceph-mon[33706]: pgmap v1259: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:37.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:36 smithi088 ceph-mon[39202]: pgmap v1259: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:37.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:36 smithi088 ceph-mon[30075]: pgmap v1259: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:39.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:38 smithi138 ceph-mon[33706]: pgmap v1260: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:38 smithi088 ceph-mon[39202]: pgmap v1260: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:39.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:38 smithi088 ceph-mon[30075]: pgmap v1260: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:41.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:40 smithi138 ceph-mon[33706]: pgmap v1261: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:40 smithi088 ceph-mon[39202]: pgmap v1261: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:41.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:40 smithi088 ceph-mon[30075]: pgmap v1261: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:41.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:41:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:41:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:42.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:42.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:41:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:41:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:42 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:42 smithi088 ceph-mon[39202]: pgmap v1262: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:42 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:43.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:42 smithi088 ceph-mon[30075]: pgmap v1262: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:43.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:42 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:42 smithi138 ceph-mon[33706]: pgmap v1262: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:45.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:44 smithi088 ceph-mon[30075]: pgmap v1263: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:44 smithi088 ceph-mon[39202]: pgmap v1263: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:45.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:44 smithi138 ceph-mon[33706]: pgmap v1263: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:47.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:46 smithi138 ceph-mon[33706]: pgmap v1264: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:46 smithi088 ceph-mon[30075]: pgmap v1264: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:46 smithi088 ceph-mon[39202]: pgmap v1264: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:41:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:48 smithi138 ceph-mon[33706]: pgmap v1265: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:48 smithi088 ceph-mon[30075]: pgmap v1265: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:48 smithi088 ceph-mon[39202]: pgmap v1265: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:51.450 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:50 smithi138 ceph-mon[33706]: pgmap v1266: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:50 smithi088 ceph-mon[30075]: pgmap v1266: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:50 smithi088 ceph-mon[39202]: pgmap v1266: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:41:51.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:41:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:41:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:41:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:41:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:53 smithi138 ceph-mon[33706]: pgmap v1267: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:53 smithi088 ceph-mon[30075]: pgmap v1267: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:53 smithi088 ceph-mon[39202]: pgmap v1267: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:55 smithi138 ceph-mon[33706]: pgmap v1268: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:55 smithi088 ceph-mon[30075]: pgmap v1268: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:55 smithi088 ceph-mon[39202]: pgmap v1268: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:56.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:56.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:57 smithi138 ceph-mon[33706]: pgmap v1269: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:57.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:57 smithi088 ceph-mon[39202]: pgmap v1269: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:57 smithi088 ceph-mon[30075]: pgmap v1269: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:41:59.275 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:41:59 smithi138 ceph-mon[33706]: pgmap v1270: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:41:59 smithi088 ceph-mon[39202]: pgmap v1270: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:41:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:41:59 smithi088 ceph-mon[30075]: pgmap v1270: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:01.461 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:01 smithi138 ceph-mon[33706]: pgmap v1271: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:01 smithi088 ceph-mon[39202]: pgmap v1271: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:01 smithi088 ceph-mon[30075]: pgmap v1271: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:01.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:42:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:42:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:42:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:42:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:03 smithi138 ceph-mon[33706]: pgmap v1272: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:03 smithi088 ceph-mon[30075]: pgmap v1272: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:03 smithi088 ceph-mon[39202]: pgmap v1272: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:05 smithi138 ceph-mon[33706]: pgmap v1273: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:05 smithi088 ceph-mon[30075]: pgmap v1273: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:05 smithi088 ceph-mon[39202]: pgmap v1273: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:07.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:07 smithi138 ceph-mon[33706]: pgmap v1274: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:07 smithi088 ceph-mon[30075]: pgmap v1274: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:07 smithi088 ceph-mon[39202]: pgmap v1274: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:09 smithi138 ceph-mon[33706]: pgmap v1275: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:09 smithi088 ceph-mon[30075]: pgmap v1275: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:09 smithi088 ceph-mon[39202]: pgmap v1275: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:11.284 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:11 smithi088 ceph-mon[39202]: pgmap v1276: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:11.284 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:11.285 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:11 smithi088 ceph-mon[30075]: pgmap v1276: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:11.285 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:11.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:11 smithi138 ceph-mon[33706]: pgmap v1276: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:11.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:11.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:42:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:42:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:42:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:42:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:13 smithi138 ceph-mon[33706]: pgmap v1277: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:13 smithi088 ceph-mon[30075]: pgmap v1277: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:13 smithi088 ceph-mon[39202]: pgmap v1277: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:15 smithi138 ceph-mon[33706]: pgmap v1278: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:15 smithi088 ceph-mon[30075]: pgmap v1278: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:15 smithi088 ceph-mon[39202]: pgmap v1278: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:17 smithi138 ceph-mon[33706]: pgmap v1279: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:17 smithi088 ceph-mon[30075]: pgmap v1279: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:17 smithi088 ceph-mon[39202]: pgmap v1279: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:19 smithi138 ceph-mon[33706]: pgmap v1280: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:19 smithi088 ceph-mon[30075]: pgmap v1280: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:19 smithi088 ceph-mon[39202]: pgmap v1280: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:21 smithi138 ceph-mon[33706]: pgmap v1281: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:21 smithi088 ceph-mon[30075]: pgmap v1281: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:21 smithi088 ceph-mon[39202]: pgmap v1281: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:21.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:42:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:42:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:42:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:42:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:23 smithi138 ceph-mon[33706]: pgmap v1282: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:23 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:42:23.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:23 smithi088 ceph-mon[30075]: pgmap v1282: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:23 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:42:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:23 smithi088 ceph-mon[39202]: pgmap v1282: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:23 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:42:24.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:42:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:42:24.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:24 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:42:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:42:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:42:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:24 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:42:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:42:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:42:24.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:24 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:42:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:25 smithi138 ceph-mon[33706]: pgmap v1283: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:25.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:25 smithi088 ceph-mon[30075]: pgmap v1283: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:25 smithi088 ceph-mon[39202]: pgmap v1283: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:26.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:26.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:26.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:27 smithi138 ceph-mon[33706]: pgmap v1284: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:27 smithi088 ceph-mon[30075]: pgmap v1284: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:27 smithi088 ceph-mon[39202]: pgmap v1284: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:29.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:29 smithi138 ceph-mon[33706]: pgmap v1285: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:29.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:29 smithi088 ceph-mon[39202]: pgmap v1285: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:29 smithi088 ceph-mon[30075]: pgmap v1285: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:31 smithi138 ceph-mon[33706]: pgmap v1286: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:31 smithi088 ceph-mon[30075]: pgmap v1286: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:31 smithi088 ceph-mon[39202]: pgmap v1286: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:42:31.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:42:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:42:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:42:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:42:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:33 smithi138 ceph-mon[33706]: pgmap v1287: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:33 smithi088 ceph-mon[30075]: pgmap v1287: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:33 smithi088 ceph-mon[39202]: pgmap v1287: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:35 smithi138 ceph-mon[33706]: pgmap v1288: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:35 smithi088 ceph-mon[30075]: pgmap v1288: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:35 smithi088 ceph-mon[39202]: pgmap v1288: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:37.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:37 smithi138 ceph-mon[33706]: pgmap v1289: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:37 smithi088 ceph-mon[30075]: pgmap v1289: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:37 smithi088 ceph-mon[39202]: pgmap v1289: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:39 smithi138 ceph-mon[33706]: pgmap v1290: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:39 smithi088 ceph-mon[39202]: pgmap v1290: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:39 smithi088 ceph-mon[30075]: pgmap v1290: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:41 smithi138 ceph-mon[33706]: pgmap v1291: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:41 smithi088 ceph-mon[39202]: pgmap v1291: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:41 smithi088 ceph-mon[30075]: pgmap v1291: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:41.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:42:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:42:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:42:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:42:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:43.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:43 smithi138 ceph-mon[33706]: pgmap v1292: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:43 smithi088 ceph-mon[30075]: pgmap v1292: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:43 smithi088 ceph-mon[39202]: pgmap v1292: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:45.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:45 smithi138 ceph-mon[33706]: pgmap v1293: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:45.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:45 smithi088 ceph-mon[39202]: pgmap v1293: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:45 smithi088 ceph-mon[30075]: pgmap v1293: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:47 smithi138 ceph-mon[33706]: pgmap v1294: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:47 smithi088 ceph-mon[39202]: pgmap v1294: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:47 smithi088 ceph-mon[30075]: pgmap v1294: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:49.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:49 smithi138 ceph-mon[33706]: pgmap v1295: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:49 smithi088 ceph-mon[39202]: pgmap v1295: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:49 smithi088 ceph-mon[30075]: pgmap v1295: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:51.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:51 smithi138 ceph-mon[33706]: pgmap v1296: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:51.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:51 smithi088 ceph-mon[39202]: pgmap v1296: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:51 smithi088 ceph-mon[30075]: pgmap v1296: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:51.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:42:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:42:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:42:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:42:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:53 smithi138 ceph-mon[33706]: pgmap v1297: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:53 smithi088 ceph-mon[39202]: pgmap v1297: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:53 smithi088 ceph-mon[30075]: pgmap v1297: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:55 smithi138 ceph-mon[33706]: pgmap v1298: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:55 smithi088 ceph-mon[39202]: pgmap v1298: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:55 smithi088 ceph-mon[30075]: pgmap v1298: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:56.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:56.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:56.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:57 smithi138 ceph-mon[33706]: pgmap v1299: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:57.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:57 smithi088 ceph-mon[39202]: pgmap v1299: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:57 smithi088 ceph-mon[30075]: pgmap v1299: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:42:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:42:59 smithi138 ceph-mon[33706]: pgmap v1300: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:59.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:42:59 smithi088 ceph-mon[30075]: pgmap v1300: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:42:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:42:59 smithi088 ceph-mon[39202]: pgmap v1300: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:01 smithi088 ceph-mon[39202]: pgmap v1301: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:01 smithi088 ceph-mon[30075]: pgmap v1301: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:01.517 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:01 smithi138 ceph-mon[33706]: pgmap v1301: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:01.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:43:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:43:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:43:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:43:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:03 smithi088 ceph-mon[30075]: pgmap v1302: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:03 smithi088 ceph-mon[39202]: pgmap v1302: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:03.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:03.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:03 smithi138 ceph-mon[33706]: pgmap v1302: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:05 smithi088 ceph-mon[30075]: pgmap v1303: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:05 smithi088 ceph-mon[39202]: pgmap v1303: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:05.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:05 smithi138 ceph-mon[33706]: pgmap v1303: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:07.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:07 smithi138 ceph-mon[33706]: pgmap v1304: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:07 smithi088 ceph-mon[30075]: pgmap v1304: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:07 smithi088 ceph-mon[39202]: pgmap v1304: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:09.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:09 smithi138 ceph-mon[33706]: pgmap v1305: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:09.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:09 smithi088 ceph-mon[39202]: pgmap v1305: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:09.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:09 smithi088 ceph-mon[30075]: pgmap v1305: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:11.528 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:11 smithi138 ceph-mon[33706]: pgmap v1306: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:11.528 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:11.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:11 smithi088 ceph-mon[39202]: pgmap v1306: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:11.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:11.534 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:11 smithi088 ceph-mon[30075]: pgmap v1306: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:11.535 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:11.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:43:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:43:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:43:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:43:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:13.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:13 smithi138 ceph-mon[33706]: pgmap v1307: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:13 smithi088 ceph-mon[39202]: pgmap v1307: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:13 smithi088 ceph-mon[30075]: pgmap v1307: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:15.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:15 smithi138 ceph-mon[33706]: pgmap v1308: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:15.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:15 smithi088 ceph-mon[39202]: pgmap v1308: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:15.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:15 smithi088 ceph-mon[30075]: pgmap v1308: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:17.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:17 smithi138 ceph-mon[33706]: pgmap v1309: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:17.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:17 smithi088 ceph-mon[39202]: pgmap v1309: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:17.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:17 smithi088 ceph-mon[30075]: pgmap v1309: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:19.224 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:43:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=cleanup t=2024-03-26T21:43:18.918711837Z level=info msg="Completed cleanup jobs" duration=3.364301ms 2024-03-26T21:43:19.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:19 smithi138 ceph-mon[33706]: pgmap v1310: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:19.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:19 smithi088 ceph-mon[39202]: pgmap v1310: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:19.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:19 smithi088 ceph-mon[30075]: pgmap v1310: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:21.539 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:21 smithi138 ceph-mon[33706]: pgmap v1311: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:21.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:21 smithi088 ceph-mon[39202]: pgmap v1311: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:21.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:21 smithi088 ceph-mon[30075]: pgmap v1311: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:21.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:43:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:43:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:43:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:43:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:23.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:23.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:23 smithi138 ceph-mon[33706]: pgmap v1312: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:23.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:23 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:43:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:23 smithi088 ceph-mon[39202]: pgmap v1312: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:23 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:43:23.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:23.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:23 smithi088 ceph-mon[30075]: pgmap v1312: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:23.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:23 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:43:24.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:43:24.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:43:24.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:24 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:43:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:43:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:43:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:24 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:43:24.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:43:24.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:43:24.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:24 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:43:25.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:25 smithi138 ceph-mon[33706]: pgmap v1313: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:25.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:25 smithi088 ceph-mon[30075]: pgmap v1313: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:25.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:25 smithi088 ceph-mon[39202]: pgmap v1313: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:26.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:26.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:26.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:27.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:27 smithi138 ceph-mon[33706]: pgmap v1314: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:27.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:27 smithi088 ceph-mon[39202]: pgmap v1314: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:27.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:27 smithi088 ceph-mon[30075]: pgmap v1314: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:29.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:29 smithi138 ceph-mon[33706]: pgmap v1315: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:29.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:29 smithi088 ceph-mon[30075]: pgmap v1315: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:29.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:29 smithi088 ceph-mon[39202]: pgmap v1315: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:31.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:31 smithi138 ceph-mon[33706]: pgmap v1316: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:31 smithi088 ceph-mon[39202]: pgmap v1316: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:31 smithi088 ceph-mon[30075]: pgmap v1316: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:31.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:43:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:43:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:43:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:43:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:33.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:33 smithi138 ceph-mon[33706]: pgmap v1317: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:33 smithi088 ceph-mon[39202]: pgmap v1317: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:33 smithi088 ceph-mon[30075]: pgmap v1317: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:35.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:35 smithi138 ceph-mon[33706]: pgmap v1318: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:35 smithi088 ceph-mon[30075]: pgmap v1318: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:35 smithi088 ceph-mon[39202]: pgmap v1318: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:37.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:37 smithi138 ceph-mon[33706]: pgmap v1319: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:37 smithi088 ceph-mon[39202]: pgmap v1319: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:37 smithi088 ceph-mon[30075]: pgmap v1319: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:39.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:39 smithi138 ceph-mon[33706]: pgmap v1320: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:39.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:39 smithi088 ceph-mon[39202]: pgmap v1320: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:39.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:39 smithi088 ceph-mon[30075]: pgmap v1320: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:41.561 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:41 smithi138 ceph-mon[33706]: pgmap v1321: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:41.561 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:41.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:41 smithi088 ceph-mon[39202]: pgmap v1321: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:41.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:41.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:41 smithi088 ceph-mon[30075]: pgmap v1321: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:41.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:41.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:43:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:43:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:43:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:43:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:43.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:43 smithi138 ceph-mon[33706]: pgmap v1322: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:43.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:43 smithi088 ceph-mon[39202]: pgmap v1322: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:43 smithi088 ceph-mon[30075]: pgmap v1322: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:45.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:45 smithi138 ceph-mon[33706]: pgmap v1323: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:45.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:45 smithi088 ceph-mon[39202]: pgmap v1323: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:45 smithi088 ceph-mon[30075]: pgmap v1323: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:47.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:47 smithi138 ceph-mon[33706]: pgmap v1324: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:47.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:47 smithi088 ceph-mon[39202]: pgmap v1324: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:47.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:47 smithi088 ceph-mon[30075]: pgmap v1324: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:48.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:48 smithi138 ceph-mon[33706]: pgmap v1325: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:48.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:48 smithi088 ceph-mon[30075]: pgmap v1325: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:48.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:48 smithi088 ceph-mon[39202]: pgmap v1325: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:51.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:50 smithi138 ceph-mon[33706]: pgmap v1326: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:51.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:50 smithi088 ceph-mon[39202]: pgmap v1326: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:51.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:50 smithi088 ceph-mon[30075]: pgmap v1326: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:51.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:43:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:43:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:43:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:43:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:53.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:53.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:52 smithi138 ceph-mon[33706]: pgmap v1327: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:52 smithi088 ceph-mon[30075]: pgmap v1327: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:52 smithi088 ceph-mon[39202]: pgmap v1327: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:55.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:54 smithi138 ceph-mon[33706]: pgmap v1328: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:54 smithi088 ceph-mon[39202]: pgmap v1328: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:55.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:54 smithi088 ceph-mon[30075]: pgmap v1328: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:57.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:56 smithi138 ceph-mon[33706]: pgmap v1329: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:57.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:56 smithi088 ceph-mon[39202]: pgmap v1329: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:56 smithi088 ceph-mon[30075]: pgmap v1329: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:43:57.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:59.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:43:58 smithi088 ceph-mon[30075]: pgmap v1330: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:59.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:43:58 smithi088 ceph-mon[39202]: pgmap v1330: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:43:58 smithi138 ceph-mon[33706]: pgmap v1330: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:01.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:00 smithi088 ceph-mon[30075]: pgmap v1331: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:01.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:00 smithi088 ceph-mon[39202]: pgmap v1331: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:01.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:00 smithi138 ceph-mon[33706]: pgmap v1331: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:01.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:44:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:44:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:44:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:44:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:02 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:03.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:02 smithi088 ceph-mon[30075]: pgmap v1332: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:02 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:03.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:02 smithi088 ceph-mon[39202]: pgmap v1332: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:02 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:02 smithi138 ceph-mon[33706]: pgmap v1332: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:04 smithi138 ceph-mon[33706]: pgmap v1333: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:05.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:04 smithi088 ceph-mon[39202]: pgmap v1333: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:04 smithi088 ceph-mon[30075]: pgmap v1333: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:07.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:06 smithi138 ceph-mon[33706]: pgmap v1334: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:06 smithi088 ceph-mon[39202]: pgmap v1334: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:06 smithi088 ceph-mon[30075]: pgmap v1334: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:08 smithi138 ceph-mon[33706]: pgmap v1335: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:08 smithi088 ceph-mon[39202]: pgmap v1335: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:08 smithi088 ceph-mon[30075]: pgmap v1335: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:11.285 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:11 smithi088 ceph-mon[39202]: pgmap v1336: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:11.286 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:11 smithi088 ceph-mon[30075]: pgmap v1336: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:11.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:11 smithi138 ceph-mon[33706]: pgmap v1336: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:11.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:44:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:44:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:12.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:44:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:44:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:13 smithi138 ceph-mon[33706]: pgmap v1337: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:13 smithi088 ceph-mon[30075]: pgmap v1337: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:13 smithi088 ceph-mon[39202]: pgmap v1337: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:15 smithi138 ceph-mon[33706]: pgmap v1338: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:15 smithi088 ceph-mon[30075]: pgmap v1338: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:15 smithi088 ceph-mon[39202]: pgmap v1338: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:17 smithi138 ceph-mon[33706]: pgmap v1339: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:17 smithi088 ceph-mon[39202]: pgmap v1339: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:17 smithi088 ceph-mon[30075]: pgmap v1339: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:19 smithi138 ceph-mon[33706]: pgmap v1340: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:19 smithi088 ceph-mon[39202]: pgmap v1340: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:19 smithi088 ceph-mon[30075]: pgmap v1340: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:21 smithi138 ceph-mon[33706]: pgmap v1341: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:21 smithi088 ceph-mon[39202]: pgmap v1341: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:21 smithi088 ceph-mon[30075]: pgmap v1341: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:21.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:44:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:44:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:44:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:44:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:23 smithi138 ceph-mon[33706]: pgmap v1342: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:23 smithi088 ceph-mon[30075]: pgmap v1342: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:23 smithi088 ceph-mon[39202]: pgmap v1342: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:24.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:24 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:44:24.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:24 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:44:24.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:24 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:44:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:25 smithi138 ceph-mon[33706]: pgmap v1343: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:44:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:44:25.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:25 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[39202]: pgmap v1343: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[30075]: pgmap v1343: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:44:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:25 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:44:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:27 smithi138 ceph-mon[33706]: pgmap v1344: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:27 smithi088 ceph-mon[30075]: pgmap v1344: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:27 smithi088 ceph-mon[39202]: pgmap v1344: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:29.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:29 smithi138 ceph-mon[33706]: pgmap v1345: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:29 smithi088 ceph-mon[39202]: pgmap v1345: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:29 smithi088 ceph-mon[30075]: pgmap v1345: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:31.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:31 smithi138 ceph-mon[33706]: pgmap v1346: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:31.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:31 smithi088 ceph-mon[30075]: pgmap v1346: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:31 smithi088 ceph-mon[39202]: pgmap v1346: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:31.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:44:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:44:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:44:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:44:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:33 smithi138 ceph-mon[33706]: pgmap v1347: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:33.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:33 smithi088 ceph-mon[30075]: pgmap v1347: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:33 smithi088 ceph-mon[39202]: pgmap v1347: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:35 smithi138 ceph-mon[33706]: pgmap v1348: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:35.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:35 smithi088 ceph-mon[30075]: pgmap v1348: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:35 smithi088 ceph-mon[39202]: pgmap v1348: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:37.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:37 smithi138 ceph-mon[33706]: pgmap v1349: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:37 smithi088 ceph-mon[39202]: pgmap v1349: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:37 smithi088 ceph-mon[30075]: pgmap v1349: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:39 smithi138 ceph-mon[33706]: pgmap v1350: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:39 smithi088 ceph-mon[39202]: pgmap v1350: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:39 smithi088 ceph-mon[30075]: pgmap v1350: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:41 smithi138 ceph-mon[33706]: pgmap v1351: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:41 smithi088 ceph-mon[30075]: pgmap v1351: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:41 smithi088 ceph-mon[39202]: pgmap v1351: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:44:41.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:44:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:44:42.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:42.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:42.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:44:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:44:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:43 smithi138 ceph-mon[33706]: pgmap v1352: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:43 smithi088 ceph-mon[39202]: pgmap v1352: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:43 smithi088 ceph-mon[30075]: pgmap v1352: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:45.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:45 smithi138 ceph-mon[33706]: pgmap v1353: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:45.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:45 smithi088 ceph-mon[39202]: pgmap v1353: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:45 smithi088 ceph-mon[30075]: pgmap v1353: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:47.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:47 smithi138 ceph-mon[33706]: pgmap v1354: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:47 smithi088 ceph-mon[39202]: pgmap v1354: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:47 smithi088 ceph-mon[30075]: pgmap v1354: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:49.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:49 smithi138 ceph-mon[33706]: pgmap v1355: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:49 smithi088 ceph-mon[39202]: pgmap v1355: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:49 smithi088 ceph-mon[30075]: pgmap v1355: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:51 smithi138 ceph-mon[33706]: pgmap v1356: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:51 smithi088 ceph-mon[39202]: pgmap v1356: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:51 smithi088 ceph-mon[30075]: pgmap v1356: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:44:51.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:44:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:44:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:44:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:44:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:53 smithi138 ceph-mon[33706]: pgmap v1357: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:53 smithi088 ceph-mon[30075]: pgmap v1357: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:53 smithi088 ceph-mon[39202]: pgmap v1357: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:55 smithi138 ceph-mon[33706]: pgmap v1358: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:55.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:55 smithi088 ceph-mon[30075]: pgmap v1358: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:55 smithi088 ceph-mon[39202]: pgmap v1358: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:57 smithi138 ceph-mon[33706]: pgmap v1359: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:57.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:57 smithi088 ceph-mon[30075]: pgmap v1359: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:57 smithi088 ceph-mon[39202]: pgmap v1359: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:44:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:44:59 smithi138 ceph-mon[33706]: pgmap v1360: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:59.482 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:44:59 smithi088 ceph-mon[30075]: pgmap v1360: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:44:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:44:59 smithi088 ceph-mon[39202]: pgmap v1360: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:01.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:01 smithi138 ceph-mon[33706]: pgmap v1361: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:01 smithi088 ceph-mon[30075]: pgmap v1361: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:01 smithi088 ceph-mon[39202]: pgmap v1361: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:01.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:45:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:45:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:45:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:45:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:03 smithi138 ceph-mon[33706]: pgmap v1362: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:03 smithi088 ceph-mon[30075]: pgmap v1362: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:03 smithi088 ceph-mon[39202]: pgmap v1362: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:05 smithi138 ceph-mon[33706]: pgmap v1363: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:05 smithi088 ceph-mon[39202]: pgmap v1363: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:05 smithi088 ceph-mon[30075]: pgmap v1363: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:07.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:07 smithi138 ceph-mon[33706]: pgmap v1364: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:07 smithi088 ceph-mon[30075]: pgmap v1364: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:07 smithi088 ceph-mon[39202]: pgmap v1364: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:09 smithi138 ceph-mon[33706]: pgmap v1365: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:09 smithi088 ceph-mon[39202]: pgmap v1365: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:09 smithi088 ceph-mon[30075]: pgmap v1365: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:11.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:11 smithi138 ceph-mon[33706]: pgmap v1366: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:11 smithi088 ceph-mon[39202]: pgmap v1366: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:11 smithi088 ceph-mon[30075]: pgmap v1366: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:11.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:45:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:45:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:12.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:45:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:45:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:13 smithi138 ceph-mon[33706]: pgmap v1367: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:13 smithi088 ceph-mon[39202]: pgmap v1367: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:13 smithi088 ceph-mon[30075]: pgmap v1367: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:15 smithi138 ceph-mon[33706]: pgmap v1368: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:15 smithi088 ceph-mon[39202]: pgmap v1368: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:15 smithi088 ceph-mon[30075]: pgmap v1368: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:17 smithi138 ceph-mon[33706]: pgmap v1369: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:17 smithi088 ceph-mon[39202]: pgmap v1369: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:17 smithi088 ceph-mon[30075]: pgmap v1369: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:19 smithi138 ceph-mon[33706]: pgmap v1370: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:19 smithi088 ceph-mon[39202]: pgmap v1370: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:19 smithi088 ceph-mon[30075]: pgmap v1370: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:21 smithi138 ceph-mon[33706]: pgmap v1371: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:21.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:21 smithi088 ceph-mon[39202]: pgmap v1371: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:21 smithi088 ceph-mon[30075]: pgmap v1371: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:21.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:45:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:45:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:45:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:45:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:23 smithi138 ceph-mon[33706]: pgmap v1372: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:23 smithi088 ceph-mon[30075]: pgmap v1372: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:23 smithi088 ceph-mon[39202]: pgmap v1372: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:25.464 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:25 smithi088 ceph-mon[39202]: pgmap v1373: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:25.464 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:25 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:45:25.465 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:25 smithi088 ceph-mon[30075]: pgmap v1373: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:25.465 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:25 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:45:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:25 smithi138 ceph-mon[33706]: pgmap v1373: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:25 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:45:27.414 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:27 smithi088 ceph-mon[39202]: pgmap v1374: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:27.414 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:27.415 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:27 smithi088 ceph-mon[30075]: pgmap v1374: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:27.415 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:27 smithi138 ceph-mon[33706]: pgmap v1374: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:28.952 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:28 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:28.953 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:28 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:28.953 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:28 smithi138 ceph-mon[33706]: pgmap v1375: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:28.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:28 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:28.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:28 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:28.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:28 smithi088 ceph-mon[30075]: pgmap v1375: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:28 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:28 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:28.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:28 smithi088 ceph-mon[39202]: pgmap v1375: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:30.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:30 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:30 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:30 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:45:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:30 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:45:30.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:30 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:45:30.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:45:30.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:45:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:45:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:30 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:45:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:31 smithi088 ceph-mon[39202]: pgmap v1376: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:31 smithi088 ceph-mon[30075]: pgmap v1376: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:31.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:31 smithi138 ceph-mon[33706]: pgmap v1376: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:31.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:45:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:45:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:45:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:45:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:33 smithi088 ceph-mon[39202]: pgmap v1377: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:33 smithi088 ceph-mon[30075]: pgmap v1377: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:33.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:33.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:33 smithi138 ceph-mon[33706]: pgmap v1377: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:35 smithi088 ceph-mon[39202]: pgmap v1378: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:35 smithi088 ceph-mon[30075]: pgmap v1378: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:35 smithi138 ceph-mon[33706]: pgmap v1378: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:37 smithi088 ceph-mon[39202]: pgmap v1379: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:37 smithi088 ceph-mon[30075]: pgmap v1379: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:37.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:37 smithi138 ceph-mon[33706]: pgmap v1379: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:39.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:39 smithi138 ceph-mon[33706]: pgmap v1380: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:39.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:39 smithi088 ceph-mon[39202]: pgmap v1380: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:39.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:39 smithi088 ceph-mon[30075]: pgmap v1380: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:41 smithi138 ceph-mon[33706]: pgmap v1381: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:41.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:41 smithi088 ceph-mon[39202]: pgmap v1381: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:41.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:41.535 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:41 smithi088 ceph-mon[30075]: pgmap v1381: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:41.535 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:41.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:45:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:45:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:45:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:45:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:43.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:43.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:43 smithi138 ceph-mon[33706]: pgmap v1382: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:43.729 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:43.730 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:43 smithi088 ceph-mon[30075]: pgmap v1382: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:43.730 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:43.730 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:43 smithi088 ceph-mon[39202]: pgmap v1382: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:45.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:45 smithi138 ceph-mon[33706]: pgmap v1383: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:45 smithi088 ceph-mon[30075]: pgmap v1383: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:45.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:45 smithi088 ceph-mon[39202]: pgmap v1383: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:47.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:47 smithi138 ceph-mon[33706]: pgmap v1384: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:47.732 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:47 smithi088 ceph-mon[30075]: pgmap v1384: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:47.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:47 smithi088 ceph-mon[39202]: pgmap v1384: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:49.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:49 smithi138 ceph-mon[33706]: pgmap v1385: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:49.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:49 smithi088 ceph-mon[30075]: pgmap v1385: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:49 smithi088 ceph-mon[39202]: pgmap v1385: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:51 smithi138 ceph-mon[33706]: pgmap v1386: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:51 smithi088 ceph-mon[30075]: pgmap v1386: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:51.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:51 smithi088 ceph-mon[39202]: pgmap v1386: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:45:51.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:45:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:45:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:45:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:45:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:53.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:53 smithi138 ceph-mon[33706]: pgmap v1387: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:53.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:53 smithi088 ceph-mon[30075]: pgmap v1387: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:53.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:53.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:53 smithi088 ceph-mon[39202]: pgmap v1387: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:55.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:55 smithi138 ceph-mon[33706]: pgmap v1388: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:55.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:55 smithi088 ceph-mon[39202]: pgmap v1388: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:55.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:55 smithi088 ceph-mon[30075]: pgmap v1388: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:56.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:56 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:56.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:56 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:56.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:56 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:57.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:57 smithi138 ceph-mon[33706]: pgmap v1389: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:57.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:57 smithi088 ceph-mon[30075]: pgmap v1389: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:57.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:57 smithi088 ceph-mon[39202]: pgmap v1389: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:45:59.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:45:59 smithi138 ceph-mon[33706]: pgmap v1390: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:59.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:45:59 smithi088 ceph-mon[30075]: pgmap v1390: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:45:59.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:45:59 smithi088 ceph-mon[39202]: pgmap v1390: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:01.526 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:01 smithi138 ceph-mon[33706]: pgmap v1391: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:01.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:01 smithi088 ceph-mon[39202]: pgmap v1391: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:01 smithi088 ceph-mon[30075]: pgmap v1391: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:01.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:46:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:46:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:46:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:46:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:03.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:03.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:03 smithi138 ceph-mon[33706]: pgmap v1392: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:03 smithi088 ceph-mon[30075]: pgmap v1392: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:03 smithi088 ceph-mon[39202]: pgmap v1392: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:05.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:05 smithi138 ceph-mon[33706]: pgmap v1393: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:05.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:05 smithi088 ceph-mon[39202]: pgmap v1393: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:05 smithi088 ceph-mon[30075]: pgmap v1393: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:07.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:07 smithi138 ceph-mon[33706]: pgmap v1394: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:07 smithi088 ceph-mon[39202]: pgmap v1394: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:07 smithi088 ceph-mon[30075]: pgmap v1394: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:09.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:09 smithi138 ceph-mon[33706]: pgmap v1395: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:09.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:09 smithi088 ceph-mon[39202]: pgmap v1395: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:09.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:09 smithi088 ceph-mon[30075]: pgmap v1395: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:11.707 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:11 smithi138 ceph-mon[33706]: pgmap v1396: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:11.707 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:11.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:11 smithi088 ceph-mon[39202]: pgmap v1396: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:11.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:11.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:11 smithi088 ceph-mon[30075]: pgmap v1396: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:11.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:11.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:46:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:46:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:46:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:46:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:13 smithi138 ceph-mon[33706]: pgmap v1397: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:13 smithi088 ceph-mon[39202]: pgmap v1397: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:13 smithi088 ceph-mon[30075]: pgmap v1397: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:15.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:15 smithi138 ceph-mon[33706]: pgmap v1398: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:15.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:15 smithi088 ceph-mon[39202]: pgmap v1398: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:15.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:15 smithi088 ceph-mon[30075]: pgmap v1398: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:17.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:17 smithi138 ceph-mon[33706]: pgmap v1399: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:17.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:17 smithi088 ceph-mon[39202]: pgmap v1399: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:17.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:17 smithi088 ceph-mon[30075]: pgmap v1399: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:19.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:19 smithi138 ceph-mon[33706]: pgmap v1400: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:19.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:19 smithi088 ceph-mon[30075]: pgmap v1400: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:19.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:19 smithi088 ceph-mon[39202]: pgmap v1400: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:21.715 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:21 smithi138 ceph-mon[33706]: pgmap v1401: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:21.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:21 smithi088 ceph-mon[39202]: pgmap v1401: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:21.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:21 smithi088 ceph-mon[30075]: pgmap v1401: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:21.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:46:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:46:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:46:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:46:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:23 smithi138 ceph-mon[33706]: pgmap v1402: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:23.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:23.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:23 smithi088 ceph-mon[30075]: pgmap v1402: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:23.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:23.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:23 smithi088 ceph-mon[39202]: pgmap v1402: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:25.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:25 smithi138 ceph-mon[33706]: pgmap v1403: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:25.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:25 smithi088 ceph-mon[39202]: pgmap v1403: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:25.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:25 smithi088 ceph-mon[30075]: pgmap v1403: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:26.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:26 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:26.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:26 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:26.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:26 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:27.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:27 smithi138 ceph-mon[33706]: pgmap v1404: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:27.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:27 smithi088 ceph-mon[39202]: pgmap v1404: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:27.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:27 smithi088 ceph-mon[30075]: pgmap v1404: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:29.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:29 smithi138 ceph-mon[33706]: pgmap v1405: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:29.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:29 smithi088 ceph-mon[30075]: pgmap v1405: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:29.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:29 smithi088 ceph-mon[39202]: pgmap v1405: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:30.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:30 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:46:30.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:30 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:46:30.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:30 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:46:31.624 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:31 smithi138 ceph-mon[33706]: pgmap v1406: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:31.624 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:31 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:46:31.625 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:31 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:46:31.625 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:31 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[39202]: pgmap v1406: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[30075]: pgmap v1406: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:46:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:31 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:46:31.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:46:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:46:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:46:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:46:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:33 smithi138 ceph-mon[33706]: pgmap v1407: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:33 smithi088 ceph-mon[39202]: pgmap v1407: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:33 smithi088 ceph-mon[30075]: pgmap v1407: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:35.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:35 smithi138 ceph-mon[33706]: pgmap v1408: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:35 smithi088 ceph-mon[39202]: pgmap v1408: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:35 smithi088 ceph-mon[30075]: pgmap v1408: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:36.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:36 smithi138 ceph-mon[33706]: pgmap v1409: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:36 smithi088 ceph-mon[39202]: pgmap v1409: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:36.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:36 smithi088 ceph-mon[30075]: pgmap v1409: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:39.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:38 smithi088 ceph-mon[39202]: pgmap v1410: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:39.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:38 smithi088 ceph-mon[30075]: pgmap v1410: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:38 smithi138 ceph-mon[33706]: pgmap v1410: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:41.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:40 smithi088 ceph-mon[39202]: pgmap v1411: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:41.234 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:40 smithi088 ceph-mon[30075]: pgmap v1411: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:40 smithi138 ceph-mon[33706]: pgmap v1411: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:42.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:41 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:42.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:46:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:46:42.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:41 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:42.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:41 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:46:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:46:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:43.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:42 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:43.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:42 smithi138 ceph-mon[33706]: pgmap v1412: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:43.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:42 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:43.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:42 smithi088 ceph-mon[30075]: pgmap v1412: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:43.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:42 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:43.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:42 smithi088 ceph-mon[39202]: pgmap v1412: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:45.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:44 smithi138 ceph-mon[33706]: pgmap v1413: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:44 smithi088 ceph-mon[39202]: pgmap v1413: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:44 smithi088 ceph-mon[30075]: pgmap v1413: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:47.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:46 smithi138 ceph-mon[33706]: pgmap v1414: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:47 smithi088 ceph-mon[39202]: pgmap v1414: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:46 smithi088 ceph-mon[30075]: pgmap v1414: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:49.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:49 smithi138 ceph-mon[33706]: pgmap v1415: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:49 smithi088 ceph-mon[30075]: pgmap v1415: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:49 smithi088 ceph-mon[39202]: pgmap v1415: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:51.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:51 smithi138 ceph-mon[33706]: pgmap v1416: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:51 smithi088 ceph-mon[39202]: pgmap v1416: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:51 smithi088 ceph-mon[30075]: pgmap v1416: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:46:52.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:46:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:46:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:46:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:46:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:53 smithi138 ceph-mon[33706]: pgmap v1417: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:53 smithi088 ceph-mon[30075]: pgmap v1417: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:53 smithi088 ceph-mon[39202]: pgmap v1417: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:55 smithi138 ceph-mon[33706]: pgmap v1418: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:55 smithi088 ceph-mon[39202]: pgmap v1418: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:55 smithi088 ceph-mon[30075]: pgmap v1418: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:57 smithi138 ceph-mon[33706]: pgmap v1419: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:57 smithi088 ceph-mon[30075]: pgmap v1419: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:57 smithi088 ceph-mon[39202]: pgmap v1419: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:46:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:46:59 smithi138 ceph-mon[33706]: pgmap v1420: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:46:59 smithi088 ceph-mon[39202]: pgmap v1420: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:46:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:46:59 smithi088 ceph-mon[30075]: pgmap v1420: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:01.276 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:01 smithi138 ceph-mon[33706]: pgmap v1421: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:01 smithi088 ceph-mon[39202]: pgmap v1421: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:01 smithi088 ceph-mon[30075]: pgmap v1421: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:02.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:47:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:47:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:47:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:47:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:03 smithi138 ceph-mon[33706]: pgmap v1422: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:03 smithi088 ceph-mon[30075]: pgmap v1422: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:03 smithi088 ceph-mon[39202]: pgmap v1422: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:05 smithi138 ceph-mon[33706]: pgmap v1423: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:05.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:05 smithi088 ceph-mon[39202]: pgmap v1423: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:05 smithi088 ceph-mon[30075]: pgmap v1423: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:07.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:07 smithi138 ceph-mon[33706]: pgmap v1424: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:07 smithi088 ceph-mon[39202]: pgmap v1424: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:07 smithi088 ceph-mon[30075]: pgmap v1424: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:09 smithi138 ceph-mon[33706]: pgmap v1425: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:09 smithi088 ceph-mon[39202]: pgmap v1425: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:09 smithi088 ceph-mon[30075]: pgmap v1425: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:11.285 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:11 smithi088 ceph-mon[30075]: pgmap v1426: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:11.285 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:11 smithi088 ceph-mon[39202]: pgmap v1426: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:11.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:11 smithi138 ceph-mon[33706]: pgmap v1426: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:12.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:47:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:47:12.465 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:12.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:47:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:47:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:13 smithi138 ceph-mon[33706]: pgmap v1427: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:13 smithi088 ceph-mon[39202]: pgmap v1427: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:13 smithi088 ceph-mon[30075]: pgmap v1427: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:15.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:15 smithi138 ceph-mon[33706]: pgmap v1428: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:15 smithi088 ceph-mon[39202]: pgmap v1428: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:15 smithi088 ceph-mon[30075]: pgmap v1428: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:17 smithi138 ceph-mon[33706]: pgmap v1429: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:17 smithi088 ceph-mon[30075]: pgmap v1429: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:17 smithi088 ceph-mon[39202]: pgmap v1429: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:19 smithi138 ceph-mon[33706]: pgmap v1430: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:19 smithi088 ceph-mon[30075]: pgmap v1430: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:19 smithi088 ceph-mon[39202]: pgmap v1430: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:21 smithi138 ceph-mon[33706]: pgmap v1431: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:21 smithi088 ceph-mon[30075]: pgmap v1431: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:21 smithi088 ceph-mon[39202]: pgmap v1431: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:22.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:47:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:47:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:47:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:47:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:23 smithi138 ceph-mon[33706]: pgmap v1432: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:23 smithi088 ceph-mon[30075]: pgmap v1432: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:23 smithi088 ceph-mon[39202]: pgmap v1432: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:25 smithi138 ceph-mon[33706]: pgmap v1433: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:25 smithi088 ceph-mon[39202]: pgmap v1433: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:25 smithi088 ceph-mon[30075]: pgmap v1433: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:27 smithi138 ceph-mon[33706]: pgmap v1434: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:27 smithi088 ceph-mon[39202]: pgmap v1434: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:27 smithi088 ceph-mon[30075]: pgmap v1434: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:29.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:29 smithi138 ceph-mon[33706]: pgmap v1435: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:29 smithi088 ceph-mon[39202]: pgmap v1435: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:29 smithi088 ceph-mon[30075]: pgmap v1435: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:31.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:31 smithi138 ceph-mon[33706]: pgmap v1436: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:31.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:31 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:47:31.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:31 smithi088 ceph-mon[39202]: pgmap v1436: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:31 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:47:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:31 smithi088 ceph-mon[30075]: pgmap v1436: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:31 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:47:32.131 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:47:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:47:32.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:47:32.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:32 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:32 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:32 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:47:32.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:32 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:47:32.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:32 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:47:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:47:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:47:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:33 smithi138 ceph-mon[33706]: pgmap v1437: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:33.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:33 smithi088 ceph-mon[39202]: pgmap v1437: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:33 smithi088 ceph-mon[30075]: pgmap v1437: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:35 smithi138 ceph-mon[33706]: pgmap v1438: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:35.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:35 smithi088 ceph-mon[39202]: pgmap v1438: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:35 smithi088 ceph-mon[30075]: pgmap v1438: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:37.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:37 smithi138 ceph-mon[33706]: pgmap v1439: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:37 smithi088 ceph-mon[30075]: pgmap v1439: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:37 smithi088 ceph-mon[39202]: pgmap v1439: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:39 smithi138 ceph-mon[33706]: pgmap v1440: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:39 smithi088 ceph-mon[30075]: pgmap v1440: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:39 smithi088 ceph-mon[39202]: pgmap v1440: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:41.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:41 smithi138 ceph-mon[33706]: pgmap v1441: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:41 smithi088 ceph-mon[39202]: pgmap v1441: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:41 smithi088 ceph-mon[30075]: pgmap v1441: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:42.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:47:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:47:42.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:42.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:42.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:47:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:47:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:43 smithi138 ceph-mon[33706]: pgmap v1442: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:43 smithi088 ceph-mon[30075]: pgmap v1442: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:43 smithi088 ceph-mon[39202]: pgmap v1442: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:45.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:45 smithi138 ceph-mon[33706]: pgmap v1443: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:45 smithi088 ceph-mon[39202]: pgmap v1443: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:45 smithi088 ceph-mon[30075]: pgmap v1443: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:47 smithi138 ceph-mon[33706]: pgmap v1444: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:47 smithi088 ceph-mon[39202]: pgmap v1444: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:47 smithi088 ceph-mon[30075]: pgmap v1444: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:49.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:49 smithi138 ceph-mon[33706]: pgmap v1445: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:49 smithi088 ceph-mon[39202]: pgmap v1445: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:49 smithi088 ceph-mon[30075]: pgmap v1445: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:51.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:51 smithi138 ceph-mon[33706]: pgmap v1446: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:51 smithi088 ceph-mon[39202]: pgmap v1446: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:51 smithi088 ceph-mon[30075]: pgmap v1446: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:52.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:47:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:47:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:47:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:47:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:53 smithi138 ceph-mon[33706]: pgmap v1447: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:53 smithi088 ceph-mon[39202]: pgmap v1447: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:53 smithi088 ceph-mon[30075]: pgmap v1447: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:55 smithi138 ceph-mon[33706]: pgmap v1448: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:55 smithi088 ceph-mon[39202]: pgmap v1448: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:55 smithi088 ceph-mon[30075]: pgmap v1448: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:57 smithi138 ceph-mon[33706]: pgmap v1449: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:57 smithi088 ceph-mon[39202]: pgmap v1449: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:57 smithi088 ceph-mon[30075]: pgmap v1449: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:47:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:47:59 smithi138 ceph-mon[33706]: pgmap v1450: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:47:59 smithi088 ceph-mon[39202]: pgmap v1450: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:47:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:47:59 smithi088 ceph-mon[30075]: pgmap v1450: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:01.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:01 smithi138 ceph-mon[33706]: pgmap v1451: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:01 smithi088 ceph-mon[39202]: pgmap v1451: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:01 smithi088 ceph-mon[30075]: pgmap v1451: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:02.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:48:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:48:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:48:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:48:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:03 smithi138 ceph-mon[33706]: pgmap v1452: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:03 smithi088 ceph-mon[30075]: pgmap v1452: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:03 smithi088 ceph-mon[39202]: pgmap v1452: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:05 smithi138 ceph-mon[33706]: pgmap v1453: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:05 smithi088 ceph-mon[39202]: pgmap v1453: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:05 smithi088 ceph-mon[30075]: pgmap v1453: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:07.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:07 smithi138 ceph-mon[33706]: pgmap v1454: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:07 smithi088 ceph-mon[39202]: pgmap v1454: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:07 smithi088 ceph-mon[30075]: pgmap v1454: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:09 smithi138 ceph-mon[33706]: pgmap v1455: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:09 smithi088 ceph-mon[30075]: pgmap v1455: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:09 smithi088 ceph-mon[39202]: pgmap v1455: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:11.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:11 smithi138 ceph-mon[33706]: pgmap v1456: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:11 smithi088 ceph-mon[30075]: pgmap v1456: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:11 smithi088 ceph-mon[39202]: pgmap v1456: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:12.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:48:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:48:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:12.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:12.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:48:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:48:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:13 smithi138 ceph-mon[33706]: pgmap v1457: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:13 smithi088 ceph-mon[30075]: pgmap v1457: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:13 smithi088 ceph-mon[39202]: pgmap v1457: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:15 smithi138 ceph-mon[33706]: pgmap v1458: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:15 smithi088 ceph-mon[39202]: pgmap v1458: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:15 smithi088 ceph-mon[30075]: pgmap v1458: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:17 smithi138 ceph-mon[33706]: pgmap v1459: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:17 smithi088 ceph-mon[30075]: pgmap v1459: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:17 smithi088 ceph-mon[39202]: pgmap v1459: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:19 smithi138 ceph-mon[33706]: pgmap v1460: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:19 smithi088 ceph-mon[30075]: pgmap v1460: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:19 smithi088 ceph-mon[39202]: pgmap v1460: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:21 smithi138 ceph-mon[33706]: pgmap v1461: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:21 smithi088 ceph-mon[30075]: pgmap v1461: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:21 smithi088 ceph-mon[39202]: pgmap v1461: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:22.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:48:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:48:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:48:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:48:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:23 smithi088 ceph-mon[39202]: pgmap v1462: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:23 smithi088 ceph-mon[30075]: pgmap v1462: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:23.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:23 smithi138 ceph-mon[33706]: pgmap v1462: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:25 smithi088 ceph-mon[30075]: pgmap v1463: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:25 smithi088 ceph-mon[39202]: pgmap v1463: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:25.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:25 smithi138 ceph-mon[33706]: pgmap v1463: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:27 smithi088 ceph-mon[39202]: pgmap v1464: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:27 smithi088 ceph-mon[30075]: pgmap v1464: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:27.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:27 smithi138 ceph-mon[33706]: pgmap v1464: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:27.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:29.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:29 smithi138 ceph-mon[33706]: pgmap v1465: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:29.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:29 smithi088 ceph-mon[30075]: pgmap v1465: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:29.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:29 smithi088 ceph-mon[39202]: pgmap v1465: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:31.526 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:31 smithi138 ceph-mon[33706]: pgmap v1466: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:31 smithi088 ceph-mon[30075]: pgmap v1466: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:31 smithi088 ceph-mon[39202]: pgmap v1466: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:32.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:48:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:48:32.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:32 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:48:32.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:32 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:48:32.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:32 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:48:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:48:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:48:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:33 smithi138 ceph-mon[33706]: pgmap v1467: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:33 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:33 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:33 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:48:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:33 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:48:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:33 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[30075]: pgmap v1467: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[39202]: pgmap v1467: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:48:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:48:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:33 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:48:35.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:35 smithi138 ceph-mon[33706]: pgmap v1468: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:35 smithi088 ceph-mon[39202]: pgmap v1468: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:35 smithi088 ceph-mon[30075]: pgmap v1468: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:37.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:37 smithi138 ceph-mon[33706]: pgmap v1469: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:37 smithi088 ceph-mon[39202]: pgmap v1469: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:37 smithi088 ceph-mon[30075]: pgmap v1469: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:39.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:39 smithi138 ceph-mon[33706]: pgmap v1470: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:39.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:39 smithi088 ceph-mon[39202]: pgmap v1470: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:39.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:39 smithi088 ceph-mon[30075]: pgmap v1470: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:41.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:41 smithi088 ceph-mon[39202]: pgmap v1471: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:41.535 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:41 smithi088 ceph-mon[30075]: pgmap v1471: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:41.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:41 smithi138 ceph-mon[33706]: pgmap v1471: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:42.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:48:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:48:42.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:42.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:48:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:48:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:43 smithi138 ceph-mon[33706]: pgmap v1472: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:43 smithi088 ceph-mon[30075]: pgmap v1472: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:43 smithi088 ceph-mon[39202]: pgmap v1472: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:45.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:45 smithi138 ceph-mon[33706]: pgmap v1473: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:45 smithi088 ceph-mon[30075]: pgmap v1473: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:45.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:45 smithi088 ceph-mon[39202]: pgmap v1473: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:47.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:47 smithi138 ceph-mon[33706]: pgmap v1474: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:47 smithi088 ceph-mon[30075]: pgmap v1474: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:47.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:47 smithi088 ceph-mon[39202]: pgmap v1474: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:49.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:49 smithi138 ceph-mon[33706]: pgmap v1475: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:49.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:49 smithi088 ceph-mon[30075]: pgmap v1475: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:49 smithi088 ceph-mon[39202]: pgmap v1475: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:51.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:51 smithi138 ceph-mon[33706]: pgmap v1476: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:51 smithi088 ceph-mon[30075]: pgmap v1476: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:51.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:51 smithi088 ceph-mon[39202]: pgmap v1476: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:48:52.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:48:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:48:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:48:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:48:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:53.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:53 smithi138 ceph-mon[33706]: pgmap v1477: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:53 smithi088 ceph-mon[30075]: pgmap v1477: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:53 smithi088 ceph-mon[39202]: pgmap v1477: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:55.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:55 smithi138 ceph-mon[33706]: pgmap v1478: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:55.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:55 smithi088 ceph-mon[30075]: pgmap v1478: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:55.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:55 smithi088 ceph-mon[39202]: pgmap v1478: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:57.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:57 smithi138 ceph-mon[33706]: pgmap v1479: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:57.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:57.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:57 smithi088 ceph-mon[30075]: pgmap v1479: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:57.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:57 smithi088 ceph-mon[39202]: pgmap v1479: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:48:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:59.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:48:59 smithi138 ceph-mon[33706]: pgmap v1480: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:59.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:48:59 smithi088 ceph-mon[30075]: pgmap v1480: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:48:59.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:48:59 smithi088 ceph-mon[39202]: pgmap v1480: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:01 smithi138 ceph-mon[33706]: pgmap v1481: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:01 smithi088 ceph-mon[30075]: pgmap v1481: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:01.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:01 smithi088 ceph-mon[39202]: pgmap v1481: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:02.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:49:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:49:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:49:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:49:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:03.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:03 smithi138 ceph-mon[33706]: pgmap v1482: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:03 smithi088 ceph-mon[39202]: pgmap v1482: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:03 smithi088 ceph-mon[30075]: pgmap v1482: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:05.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:05 smithi138 ceph-mon[33706]: pgmap v1483: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:05.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:05 smithi088 ceph-mon[30075]: pgmap v1483: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:05 smithi088 ceph-mon[39202]: pgmap v1483: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:07.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:07 smithi138 ceph-mon[33706]: pgmap v1484: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:07 smithi088 ceph-mon[30075]: pgmap v1484: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:07 smithi088 ceph-mon[39202]: pgmap v1484: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:09.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:09 smithi138 ceph-mon[33706]: pgmap v1485: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:09.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:09 smithi088 ceph-mon[30075]: pgmap v1485: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:09.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:09 smithi088 ceph-mon[39202]: pgmap v1485: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:11.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:11 smithi138 ceph-mon[33706]: pgmap v1486: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:11.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:11 smithi088 ceph-mon[30075]: pgmap v1486: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:11.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:11 smithi088 ceph-mon[39202]: pgmap v1486: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:12.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:49:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:49:12.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:12.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:12.735 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:49:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:49:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:13.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:13 smithi138 ceph-mon[33706]: pgmap v1487: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:13.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:13 smithi088 ceph-mon[30075]: pgmap v1487: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:13 smithi088 ceph-mon[39202]: pgmap v1487: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:15.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:15 smithi138 ceph-mon[33706]: pgmap v1488: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:15.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:15 smithi088 ceph-mon[30075]: pgmap v1488: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:15.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:15 smithi088 ceph-mon[39202]: pgmap v1488: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:17.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:17 smithi138 ceph-mon[33706]: pgmap v1489: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:17.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:17 smithi088 ceph-mon[30075]: pgmap v1489: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:17.769 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:17 smithi088 ceph-mon[39202]: pgmap v1489: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:19.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:19 smithi138 ceph-mon[33706]: pgmap v1490: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:19.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:19 smithi088 ceph-mon[30075]: pgmap v1490: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:19.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:19 smithi088 ceph-mon[39202]: pgmap v1490: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:21.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:21 smithi138 ceph-mon[33706]: pgmap v1491: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:21.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:21 smithi088 ceph-mon[30075]: pgmap v1491: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:21.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:21 smithi088 ceph-mon[39202]: pgmap v1491: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:22.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:49:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:49:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:49:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:49:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:23.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:23 smithi138 ceph-mon[33706]: pgmap v1492: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:23.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:23.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:23 smithi088 ceph-mon[30075]: pgmap v1492: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:23 smithi088 ceph-mon[39202]: pgmap v1492: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:25.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:25 smithi138 ceph-mon[33706]: pgmap v1493: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:25.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:25 smithi088 ceph-mon[30075]: pgmap v1493: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:25.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:25 smithi088 ceph-mon[39202]: pgmap v1493: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:27.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:27 smithi138 ceph-mon[33706]: pgmap v1494: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:27.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:27.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:27 smithi088 ceph-mon[30075]: pgmap v1494: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:27.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:27.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:27 smithi088 ceph-mon[39202]: pgmap v1494: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:27.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:29.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:29 smithi138 ceph-mon[33706]: pgmap v1495: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:29.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:29 smithi088 ceph-mon[30075]: pgmap v1495: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:29.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:29 smithi088 ceph-mon[39202]: pgmap v1495: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:31.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:31 smithi138 ceph-mon[33706]: pgmap v1496: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:31 smithi088 ceph-mon[30075]: pgmap v1496: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:31 smithi088 ceph-mon[39202]: pgmap v1496: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:32.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:49:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:49:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:49:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:49:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:33.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:33 smithi138 ceph-mon[33706]: pgmap v1497: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:33 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:49:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:33 smithi088 ceph-mon[30075]: pgmap v1497: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:33 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:49:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:33 smithi088 ceph-mon[39202]: pgmap v1497: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:33 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:49:35.661 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:35 smithi088 ceph-mon[30075]: pgmap v1498: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:35.661 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:35 smithi088 ceph-mon[39202]: pgmap v1498: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:35.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:35 smithi138 ceph-mon[33706]: pgmap v1498: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:37 smithi138 ceph-mon[33706]: pgmap v1499: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:37 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:37 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:37 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:49:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:37 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:49:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:37 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[39202]: pgmap v1499: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:49:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:49:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[30075]: pgmap v1499: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:37.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:37.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:49:37.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:49:37.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:37 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:49:39.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:39 smithi138 ceph-mon[33706]: pgmap v1500: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:39.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:39 smithi088 ceph-mon[39202]: pgmap v1500: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:39.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:39 smithi088 ceph-mon[30075]: pgmap v1500: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:41.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:41 smithi138 ceph-mon[33706]: pgmap v1501: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:41.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:41 smithi088 ceph-mon[39202]: pgmap v1501: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:41.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:41 smithi088 ceph-mon[30075]: pgmap v1501: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:42.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:49:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:49:42.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:42.732 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:49:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:49:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:43 smithi138 ceph-mon[33706]: pgmap v1502: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:43 smithi088 ceph-mon[30075]: pgmap v1502: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:43 smithi088 ceph-mon[39202]: pgmap v1502: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:44.723 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:44 smithi138 ceph-mon[33706]: pgmap v1503: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:44.732 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:44 smithi088 ceph-mon[30075]: pgmap v1503: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:44.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:44 smithi088 ceph-mon[39202]: pgmap v1503: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:47.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:47 smithi138 ceph-mon[33706]: pgmap v1504: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:47 smithi088 ceph-mon[30075]: pgmap v1504: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:47.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:47 smithi088 ceph-mon[39202]: pgmap v1504: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:49.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:49 smithi138 ceph-mon[33706]: pgmap v1505: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:49 smithi088 ceph-mon[30075]: pgmap v1505: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:49 smithi088 ceph-mon[39202]: pgmap v1505: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:51 smithi138 ceph-mon[33706]: pgmap v1506: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:51 smithi088 ceph-mon[30075]: pgmap v1506: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:51 smithi088 ceph-mon[39202]: pgmap v1506: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:49:52.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:49:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:49:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:49:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:49:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:53.233 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:53 smithi088 ceph-mon[30075]: pgmap v1507: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:53.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:53 smithi088 ceph-mon[39202]: pgmap v1507: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:53 smithi138 ceph-mon[33706]: pgmap v1507: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:55.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:55 smithi138 ceph-mon[33706]: pgmap v1508: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:55 smithi088 ceph-mon[30075]: pgmap v1508: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:55 smithi088 ceph-mon[39202]: pgmap v1508: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:57 smithi138 ceph-mon[33706]: pgmap v1509: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:57 smithi088 ceph-mon[30075]: pgmap v1509: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:57 smithi088 ceph-mon[39202]: pgmap v1509: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:49:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:49:59 smithi138 ceph-mon[33706]: pgmap v1510: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:49:59 smithi088 ceph-mon[30075]: pgmap v1510: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:49:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:49:59 smithi088 ceph-mon[39202]: pgmap v1510: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:00.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:00 smithi138 ceph-mon[33706]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:50:00.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:00 smithi088 ceph-mon[30075]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:50:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:00 smithi088 ceph-mon[39202]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T21:50:01.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:01 smithi138 ceph-mon[33706]: pgmap v1511: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:01 smithi088 ceph-mon[30075]: pgmap v1511: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:01 smithi088 ceph-mon[39202]: pgmap v1511: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:02.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:50:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:50:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:50:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:50:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:03 smithi138 ceph-mon[33706]: pgmap v1512: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:03 smithi088 ceph-mon[30075]: pgmap v1512: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:03 smithi088 ceph-mon[39202]: pgmap v1512: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:05 smithi138 ceph-mon[33706]: pgmap v1513: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:05 smithi088 ceph-mon[30075]: pgmap v1513: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:05 smithi088 ceph-mon[39202]: pgmap v1513: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:07.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:07 smithi138 ceph-mon[33706]: pgmap v1514: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:07 smithi088 ceph-mon[39202]: pgmap v1514: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:07 smithi088 ceph-mon[30075]: pgmap v1514: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:09.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:09 smithi138 ceph-mon[33706]: pgmap v1515: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:09 smithi088 ceph-mon[30075]: pgmap v1515: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:09 smithi088 ceph-mon[39202]: pgmap v1515: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:11.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:11 smithi138 ceph-mon[33706]: pgmap v1516: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:11 smithi088 ceph-mon[39202]: pgmap v1516: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:11 smithi088 ceph-mon[30075]: pgmap v1516: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:12.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:12.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:50:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:50:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:12.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:50:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:50:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:13 smithi138 ceph-mon[33706]: pgmap v1517: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:13 smithi088 ceph-mon[39202]: pgmap v1517: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:13 smithi088 ceph-mon[30075]: pgmap v1517: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:15 smithi138 ceph-mon[33706]: pgmap v1518: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:15.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:15 smithi088 ceph-mon[39202]: pgmap v1518: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:15 smithi088 ceph-mon[30075]: pgmap v1518: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:17 smithi138 ceph-mon[33706]: pgmap v1519: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:17 smithi088 ceph-mon[30075]: pgmap v1519: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:17 smithi088 ceph-mon[39202]: pgmap v1519: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:19.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:19 smithi138 ceph-mon[33706]: pgmap v1520: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:19 smithi088 ceph-mon[30075]: pgmap v1520: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:19 smithi088 ceph-mon[39202]: pgmap v1520: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:21 smithi138 ceph-mon[33706]: pgmap v1521: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:21 smithi088 ceph-mon[30075]: pgmap v1521: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:21 smithi088 ceph-mon[39202]: pgmap v1521: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:22.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:50:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:50:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:50:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:50:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:23.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:23.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:23 smithi138 ceph-mon[33706]: pgmap v1522: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:23 smithi088 ceph-mon[39202]: pgmap v1522: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:23 smithi088 ceph-mon[30075]: pgmap v1522: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:25.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:25 smithi138 ceph-mon[33706]: pgmap v1523: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:25 smithi088 ceph-mon[39202]: pgmap v1523: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:25.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:25 smithi088 ceph-mon[30075]: pgmap v1523: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:27.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:27 smithi138 ceph-mon[33706]: pgmap v1524: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:27.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:27 smithi088 ceph-mon[30075]: pgmap v1524: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:27.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:27 smithi088 ceph-mon[39202]: pgmap v1524: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:27.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:29.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:29 smithi138 ceph-mon[33706]: pgmap v1525: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:29.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:29 smithi088 ceph-mon[39202]: pgmap v1525: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:29.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:29 smithi088 ceph-mon[30075]: pgmap v1525: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:31.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:31 smithi138 ceph-mon[33706]: pgmap v1526: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:31 smithi088 ceph-mon[39202]: pgmap v1526: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:31.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:31 smithi088 ceph-mon[30075]: pgmap v1526: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:32.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:50:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:50:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:50:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:50:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:33.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:33.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:33 smithi138 ceph-mon[33706]: pgmap v1527: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:33 smithi088 ceph-mon[39202]: pgmap v1527: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:33.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:33 smithi088 ceph-mon[30075]: pgmap v1527: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:35.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:35 smithi138 ceph-mon[33706]: pgmap v1528: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:35 smithi088 ceph-mon[39202]: pgmap v1528: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:35.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:35 smithi088 ceph-mon[30075]: pgmap v1528: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:37.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:37 smithi138 ceph-mon[33706]: pgmap v1529: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:37.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:37 smithi088 ceph-mon[30075]: pgmap v1529: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:37 smithi088 ceph-mon[39202]: pgmap v1529: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:38.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:38 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:50:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:38 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:50:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:38 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:50:38.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:38 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:50:38.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:50:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:50:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:50:38.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:50:38.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:50:38.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:50:38.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:50:38.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:38 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:50:39.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:39 smithi138 ceph-mon[33706]: pgmap v1530: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:39.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:39 smithi088 ceph-mon[30075]: pgmap v1530: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:39.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:39 smithi088 ceph-mon[39202]: pgmap v1530: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:41.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:41 smithi138 ceph-mon[33706]: pgmap v1531: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:41.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:41 smithi088 ceph-mon[39202]: pgmap v1531: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:41.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:41 smithi088 ceph-mon[30075]: pgmap v1531: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:42.223 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:42.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:50:41 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:50:42.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:42.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:42.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:50:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:50:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:43.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:43 smithi138 ceph-mon[33706]: pgmap v1532: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:43 smithi088 ceph-mon[39202]: pgmap v1532: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:43.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:43 smithi088 ceph-mon[30075]: pgmap v1532: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:45.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:45 smithi138 ceph-mon[33706]: pgmap v1533: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:45.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:45 smithi088 ceph-mon[39202]: pgmap v1533: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:45.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:45 smithi088 ceph-mon[30075]: pgmap v1533: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:47.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:47 smithi138 ceph-mon[33706]: pgmap v1534: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:47.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:47 smithi088 ceph-mon[39202]: pgmap v1534: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:47.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:47 smithi088 ceph-mon[30075]: pgmap v1534: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:49.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:49 smithi138 ceph-mon[33706]: pgmap v1535: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:49.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:49 smithi088 ceph-mon[39202]: pgmap v1535: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:49.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:49 smithi088 ceph-mon[30075]: pgmap v1535: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:51.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:51 smithi138 ceph-mon[33706]: pgmap v1536: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:51.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:51 smithi088 ceph-mon[39202]: pgmap v1536: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:51.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:51 smithi088 ceph-mon[30075]: pgmap v1536: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:50:52.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:50:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:50:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:50:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:50:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:53.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:53.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:53 smithi138 ceph-mon[33706]: pgmap v1537: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:53.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:53 smithi088 ceph-mon[39202]: pgmap v1537: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:53.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:53 smithi088 ceph-mon[30075]: pgmap v1537: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:55.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:55 smithi138 ceph-mon[33706]: pgmap v1538: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:55.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:55 smithi088 ceph-mon[30075]: pgmap v1538: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:55 smithi088 ceph-mon[39202]: pgmap v1538: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:57.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:57 smithi138 ceph-mon[33706]: pgmap v1539: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:57.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:57 smithi088 ceph-mon[39202]: pgmap v1539: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:57.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:57 smithi088 ceph-mon[30075]: pgmap v1539: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:50:57.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:59.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:50:59 smithi138 ceph-mon[33706]: pgmap v1540: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:50:59 smithi088 ceph-mon[39202]: pgmap v1540: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:50:59.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:50:59 smithi088 ceph-mon[30075]: pgmap v1540: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:01.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:01 smithi138 ceph-mon[33706]: pgmap v1541: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:01.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:01 smithi088 ceph-mon[30075]: pgmap v1541: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:01 smithi088 ceph-mon[39202]: pgmap v1541: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:02.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:51:01 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:51:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:51:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:51:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:03.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:03.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:03 smithi138 ceph-mon[33706]: pgmap v1542: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:03.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:03 smithi088 ceph-mon[39202]: pgmap v1542: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:03.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:03 smithi088 ceph-mon[30075]: pgmap v1542: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:05.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:05 smithi138 ceph-mon[33706]: pgmap v1543: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:05 smithi088 ceph-mon[39202]: pgmap v1543: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:05.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:05 smithi088 ceph-mon[30075]: pgmap v1543: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:07.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:07 smithi138 ceph-mon[33706]: pgmap v1544: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:07.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:07 smithi088 ceph-mon[39202]: pgmap v1544: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:07.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:07 smithi088 ceph-mon[30075]: pgmap v1544: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:09.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:09 smithi138 ceph-mon[33706]: pgmap v1545: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:09 smithi088 ceph-mon[39202]: pgmap v1545: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:09.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:09 smithi088 ceph-mon[30075]: pgmap v1545: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:11.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:11 smithi138 ceph-mon[33706]: pgmap v1546: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:11 smithi088 ceph-mon[39202]: pgmap v1546: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:11.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:11 smithi088 ceph-mon[30075]: pgmap v1546: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:12.224 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:12.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:51:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:51:12.466 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:12.466 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:12.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:51:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:51:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:13.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:13.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:13 smithi138 ceph-mon[33706]: pgmap v1547: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:13 smithi088 ceph-mon[39202]: pgmap v1547: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:13.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:13 smithi088 ceph-mon[30075]: pgmap v1547: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:15.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:15 smithi138 ceph-mon[33706]: pgmap v1548: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:15.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:15 smithi088 ceph-mon[39202]: pgmap v1548: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:15.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:15 smithi088 ceph-mon[30075]: pgmap v1548: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:17.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:17 smithi138 ceph-mon[33706]: pgmap v1549: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:17.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:17 smithi088 ceph-mon[39202]: pgmap v1549: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:17.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:17 smithi088 ceph-mon[30075]: pgmap v1549: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:19.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:19 smithi138 ceph-mon[33706]: pgmap v1550: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:19.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:19 smithi088 ceph-mon[30075]: pgmap v1550: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:19.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:19 smithi088 ceph-mon[39202]: pgmap v1550: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:21.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:21 smithi138 ceph-mon[33706]: pgmap v1551: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:21.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:21 smithi088 ceph-mon[39202]: pgmap v1551: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:21.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:21 smithi088 ceph-mon[30075]: pgmap v1551: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:22.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:51:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:51:22.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:51:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:51:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:23.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:23 smithi088 ceph-mon[39202]: pgmap v1552: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:23.483 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:23 smithi088 ceph-mon[30075]: pgmap v1552: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:23 smithi138 ceph-mon[33706]: pgmap v1552: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:25.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:25 smithi138 ceph-mon[33706]: pgmap v1553: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:25.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:25 smithi088 ceph-mon[39202]: pgmap v1553: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:25.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:25 smithi088 ceph-mon[30075]: pgmap v1553: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:27.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:27 smithi138 ceph-mon[33706]: pgmap v1554: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:27.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:27.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:27 smithi088 ceph-mon[30075]: pgmap v1554: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:27.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:27.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:27 smithi088 ceph-mon[39202]: pgmap v1554: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:27.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:29.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:29 smithi138 ceph-mon[33706]: pgmap v1555: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:29.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:29 smithi088 ceph-mon[39202]: pgmap v1555: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:29.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:29 smithi088 ceph-mon[30075]: pgmap v1555: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:31.526 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:31 smithi138 ceph-mon[33706]: pgmap v1556: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:31.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:31 smithi088 ceph-mon[39202]: pgmap v1556: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:31.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:31 smithi088 ceph-mon[30075]: pgmap v1556: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:32.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:51:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:51:32.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:51:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:51:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:33 smithi138 ceph-mon[33706]: pgmap v1557: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:33.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:33 smithi088 ceph-mon[30075]: pgmap v1557: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:33 smithi088 ceph-mon[39202]: pgmap v1557: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:35 smithi138 ceph-mon[33706]: pgmap v1558: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:35 smithi088 ceph-mon[39202]: pgmap v1558: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:35.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:35 smithi088 ceph-mon[30075]: pgmap v1558: 260 pgs: 260 active+clean; 582 KiB data, 724 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:37 smithi138 ceph-mon[33706]: pgmap v1559: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:37 smithi088 ceph-mon[39202]: pgmap v1559: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:37.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:37 smithi088 ceph-mon[30075]: pgmap v1559: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:38.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:38 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:51:38.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:38 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:51:38.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:38 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:51:39.374 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:39 smithi138 ceph-mon[33706]: pgmap v1560: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:39.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:39 smithi088 ceph-mon[39202]: pgmap v1560: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:39.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:39 smithi088 ceph-mon[30075]: pgmap v1560: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:41.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:41 smithi088 ceph-mon[39202]: pgmap v1561: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:41.536 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:41 smithi088 ceph-mon[30075]: pgmap v1561: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:41.638 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:41 smithi138 ceph-mon[33706]: pgmap v1561: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:42.277 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:51:42 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:51:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:42 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:42 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:51:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:51:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:42 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:51:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:51:42.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:51:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:51:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:51:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:51:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:51:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:43 smithi138 ceph-mon[33706]: pgmap v1562: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:43.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:43 smithi088 ceph-mon[30075]: pgmap v1562: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:43.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:43 smithi088 ceph-mon[39202]: pgmap v1562: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:45.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:45 smithi138 ceph-mon[33706]: pgmap v1563: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:45.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:45 smithi088 ceph-mon[30075]: pgmap v1563: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:45.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:45 smithi088 ceph-mon[39202]: pgmap v1563: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:47.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:47 smithi138 ceph-mon[33706]: pgmap v1564: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:47.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:47 smithi088 ceph-mon[39202]: pgmap v1564: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:47.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:47 smithi088 ceph-mon[30075]: pgmap v1564: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T21:51:49.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:49 smithi138 ceph-mon[33706]: pgmap v1565: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:49.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:49 smithi088 ceph-mon[30075]: pgmap v1565: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:49 smithi088 ceph-mon[39202]: pgmap v1565: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:51.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:51 smithi138 ceph-mon[33706]: pgmap v1566: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:51.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:51 smithi088 ceph-mon[39202]: pgmap v1566: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:51.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:51 smithi088 ceph-mon[30075]: pgmap v1566: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:52.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:51:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:51:52.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:51:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:51:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:53.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:53 smithi138 ceph-mon[33706]: pgmap v1567: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:53.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:53 smithi088 ceph-mon[39202]: pgmap v1567: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:53.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:53 smithi088 ceph-mon[30075]: pgmap v1567: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:55.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:55 smithi138 ceph-mon[33706]: pgmap v1568: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:55.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:55 smithi088 ceph-mon[39202]: pgmap v1568: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:55.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:55 smithi088 ceph-mon[30075]: pgmap v1568: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:57.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:57 smithi138 ceph-mon[33706]: pgmap v1569: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:57.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:57 smithi088 ceph-mon[39202]: pgmap v1569: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:57.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:57.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:57 smithi088 ceph-mon[30075]: pgmap v1569: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:51:57.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:59.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:51:59 smithi138 ceph-mon[33706]: pgmap v1570: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:59.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:51:59 smithi088 ceph-mon[39202]: pgmap v1570: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:51:59.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:51:59 smithi088 ceph-mon[30075]: pgmap v1570: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:01.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:01 smithi138 ceph-mon[33706]: pgmap v1571: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:01.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:01 smithi088 ceph-mon[39202]: pgmap v1571: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:01.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:01 smithi088 ceph-mon[30075]: pgmap v1571: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:02.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:52:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:52:02.733 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:52:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:52:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:03.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:03 smithi138 ceph-mon[33706]: pgmap v1572: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:03.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:03 smithi088 ceph-mon[39202]: pgmap v1572: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:03.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:03.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:03 smithi088 ceph-mon[30075]: pgmap v1572: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:05.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:05 smithi138 ceph-mon[33706]: pgmap v1573: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:05.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:05 smithi088 ceph-mon[39202]: pgmap v1573: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:05.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:05 smithi088 ceph-mon[30075]: pgmap v1573: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:07.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:07 smithi138 ceph-mon[33706]: pgmap v1574: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:07 smithi088 ceph-mon[39202]: pgmap v1574: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:07.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:07 smithi088 ceph-mon[30075]: pgmap v1574: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:09.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:09 smithi138 ceph-mon[33706]: pgmap v1575: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:09.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:09 smithi088 ceph-mon[39202]: pgmap v1575: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:09.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:09 smithi088 ceph-mon[30075]: pgmap v1575: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:11.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:11 smithi138 ceph-mon[33706]: pgmap v1576: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:11.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:11 smithi088 ceph-mon[39202]: pgmap v1576: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:11.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:11 smithi088 ceph-mon[30075]: pgmap v1576: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:12.339 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:52:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:52:12.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:12.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:12.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:12.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:52:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:52:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:13.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:13 smithi138 ceph-mon[33706]: pgmap v1577: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:13.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:13 smithi088 ceph-mon[39202]: pgmap v1577: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:13.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:13.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:13 smithi088 ceph-mon[30075]: pgmap v1577: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:15.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:15 smithi138 ceph-mon[33706]: pgmap v1578: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:15.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:15 smithi088 ceph-mon[39202]: pgmap v1578: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:15.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:15 smithi088 ceph-mon[30075]: pgmap v1578: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:17.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:17 smithi138 ceph-mon[33706]: pgmap v1579: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:17.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:17 smithi088 ceph-mon[39202]: pgmap v1579: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:17.733 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:17 smithi088 ceph-mon[30075]: pgmap v1579: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:19.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:19 smithi138 ceph-mon[33706]: pgmap v1580: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:19.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:19 smithi088 ceph-mon[39202]: pgmap v1580: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:19.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:19 smithi088 ceph-mon[30075]: pgmap v1580: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:21.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:21 smithi138 ceph-mon[33706]: pgmap v1581: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:21.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:21 smithi088 ceph-mon[39202]: pgmap v1581: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:21.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:21 smithi088 ceph-mon[30075]: pgmap v1581: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T21:52:22.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:52:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:52:22.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:52:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:52:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:23.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:23 smithi138 ceph-mon[33706]: pgmap v1582: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:23 smithi088 ceph-mon[39202]: pgmap v1582: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:23.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:23.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:23 smithi088 ceph-mon[30075]: pgmap v1582: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:25.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:25 smithi138 ceph-mon[33706]: pgmap v1583: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:25.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:25 smithi088 ceph-mon[39202]: pgmap v1583: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:25.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:25 smithi088 ceph-mon[30075]: pgmap v1583: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:27.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:27 smithi138 ceph-mon[33706]: pgmap v1584: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:27.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:27.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:27 smithi088 ceph-mon[39202]: pgmap v1584: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:27.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:27.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:27 smithi088 ceph-mon[30075]: pgmap v1584: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:27.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:29.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:29 smithi138 ceph-mon[33706]: pgmap v1585: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:29 smithi088 ceph-mon[39202]: pgmap v1585: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:29.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:29 smithi088 ceph-mon[30075]: pgmap v1585: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:31.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:31 smithi138 ceph-mon[33706]: pgmap v1586: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:31.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:31 smithi088 ceph-mon[39202]: pgmap v1586: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:31.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:31 smithi088 ceph-mon[30075]: pgmap v1586: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:32.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:52:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:52:32.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:52:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:52:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:33.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:33.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:33 smithi138 ceph-mon[33706]: pgmap v1587: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:33.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:33 smithi088 ceph-mon[39202]: pgmap v1587: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:33.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:33.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:33 smithi088 ceph-mon[30075]: pgmap v1587: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:35.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:35 smithi138 ceph-mon[33706]: pgmap v1588: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:35.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:35 smithi088 ceph-mon[39202]: pgmap v1588: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:35.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:35 smithi088 ceph-mon[30075]: pgmap v1588: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:37.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:37 smithi138 ceph-mon[33706]: pgmap v1589: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:37 smithi088 ceph-mon[39202]: pgmap v1589: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:37.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:37 smithi088 ceph-mon[30075]: pgmap v1589: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:39.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:39 smithi138 ceph-mon[33706]: pgmap v1590: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:39.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:39 smithi088 ceph-mon[39202]: pgmap v1590: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:39.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:39 smithi088 ceph-mon[30075]: pgmap v1590: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:41.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:41 smithi138 ceph-mon[33706]: pgmap v1591: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:41.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:41 smithi088 ceph-mon[39202]: pgmap v1591: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:41.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:41 smithi088 ceph-mon[30075]: pgmap v1591: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:42.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:52:42 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:52:42.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:42.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:52:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:52:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:42.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:52:42.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:52:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:52:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:43.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: pgmap v1592: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:52:43.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:43 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: pgmap v1592: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:52:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: pgmap v1592: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:52:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:43 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:52:44.724 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:44 smithi138 ceph-mon[33706]: pgmap v1593: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:44 smithi088 ceph-mon[39202]: pgmap v1593: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:44.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:44 smithi088 ceph-mon[30075]: pgmap v1593: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:47 smithi138 ceph-mon[33706]: pgmap v1594: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:47.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:47 smithi088 ceph-mon[30075]: pgmap v1594: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:47 smithi088 ceph-mon[39202]: pgmap v1594: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:49.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:49 smithi138 ceph-mon[33706]: pgmap v1595: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:49 smithi088 ceph-mon[39202]: pgmap v1595: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:49 smithi088 ceph-mon[30075]: pgmap v1595: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:51.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:51 smithi138 ceph-mon[33706]: pgmap v1596: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:51 smithi088 ceph-mon[39202]: pgmap v1596: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:51.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:51 smithi088 ceph-mon[30075]: pgmap v1596: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:52.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:52:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:52:52.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:52:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:52:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:53.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:53 smithi138 ceph-mon[33706]: pgmap v1597: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:53 smithi088 ceph-mon[39202]: pgmap v1597: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:53.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:53 smithi088 ceph-mon[30075]: pgmap v1597: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:55.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:55 smithi138 ceph-mon[33706]: pgmap v1598: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:55 smithi088 ceph-mon[39202]: pgmap v1598: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:55.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:55 smithi088 ceph-mon[30075]: pgmap v1598: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:57.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:57 smithi138 ceph-mon[33706]: pgmap v1599: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:57.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:57 smithi088 ceph-mon[39202]: pgmap v1599: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:57 smithi088 ceph-mon[30075]: pgmap v1599: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:57.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:59.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:52:59 smithi138 ceph-mon[33706]: pgmap v1600: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:59.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:52:59 smithi088 ceph-mon[30075]: pgmap v1600: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:52:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:52:59 smithi088 ceph-mon[39202]: pgmap v1600: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:01.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:01 smithi138 ceph-mon[33706]: pgmap v1601: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:01.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:01 smithi088 ceph-mon[30075]: pgmap v1601: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:01 smithi088 ceph-mon[39202]: pgmap v1601: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:02.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:53:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:53:02.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:53:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:53:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:03.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:03.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:03 smithi138 ceph-mon[33706]: pgmap v1602: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:03.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:03.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:03 smithi088 ceph-mon[30075]: pgmap v1602: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:03.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:03.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:03 smithi088 ceph-mon[39202]: pgmap v1602: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:05.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:05 smithi138 ceph-mon[33706]: pgmap v1603: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:05.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:05 smithi088 ceph-mon[30075]: pgmap v1603: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:05 smithi088 ceph-mon[39202]: pgmap v1603: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:07.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:07 smithi138 ceph-mon[33706]: pgmap v1604: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:07 smithi088 ceph-mon[39202]: pgmap v1604: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:07.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:07 smithi088 ceph-mon[30075]: pgmap v1604: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:09.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:09 smithi138 ceph-mon[33706]: pgmap v1605: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:09.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:09 smithi088 ceph-mon[30075]: pgmap v1605: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:09.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:09 smithi088 ceph-mon[39202]: pgmap v1605: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:11.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:11 smithi138 ceph-mon[33706]: pgmap v1606: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:11 smithi088 ceph-mon[39202]: pgmap v1606: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:11.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:11 smithi088 ceph-mon[30075]: pgmap v1606: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:12.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:12.467 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:12.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:12.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:53:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:53:12.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:53:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:53:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:13.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:13.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:13 smithi138 ceph-mon[33706]: pgmap v1607: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:13 smithi088 ceph-mon[39202]: pgmap v1607: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:13.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:13.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:13 smithi088 ceph-mon[30075]: pgmap v1607: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:15.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:15 smithi138 ceph-mon[33706]: pgmap v1608: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:15.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:15 smithi088 ceph-mon[30075]: pgmap v1608: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:15 smithi088 ceph-mon[39202]: pgmap v1608: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:17.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:17 smithi138 ceph-mon[33706]: pgmap v1609: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:17.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:17 smithi088 ceph-mon[30075]: pgmap v1609: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:17.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:17 smithi088 ceph-mon[39202]: pgmap v1609: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:19.174 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 21:53:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=cleanup t=2024-03-26T21:53:18.919592106Z level=info msg="Completed cleanup jobs" duration=4.611818ms 2024-03-26T21:53:19.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:19 smithi138 ceph-mon[33706]: pgmap v1610: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:19.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:19 smithi088 ceph-mon[30075]: pgmap v1610: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:19 smithi088 ceph-mon[39202]: pgmap v1610: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:21.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:21 smithi138 ceph-mon[33706]: pgmap v1611: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:21.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:21 smithi088 ceph-mon[39202]: pgmap v1611: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:21.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:21 smithi088 ceph-mon[30075]: pgmap v1611: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:22.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:53:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:53:22.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:53:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:53:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:23.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:23 smithi138 ceph-mon[33706]: pgmap v1612: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:23.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:23 smithi088 ceph-mon[39202]: pgmap v1612: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:23.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:23 smithi088 ceph-mon[30075]: pgmap v1612: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:23.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:25.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:25 smithi138 ceph-mon[33706]: pgmap v1613: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:25 smithi088 ceph-mon[39202]: pgmap v1613: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:25.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:25 smithi088 ceph-mon[30075]: pgmap v1613: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:27 smithi138 ceph-mon[33706]: pgmap v1614: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:27.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:27 smithi088 ceph-mon[39202]: pgmap v1614: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:27 smithi088 ceph-mon[30075]: pgmap v1614: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:27.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:29.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:29 smithi138 ceph-mon[33706]: pgmap v1615: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:29.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:29 smithi088 ceph-mon[30075]: pgmap v1615: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:29 smithi088 ceph-mon[39202]: pgmap v1615: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:31.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:31 smithi138 ceph-mon[33706]: pgmap v1616: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:31 smithi088 ceph-mon[39202]: pgmap v1616: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:31.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:31 smithi088 ceph-mon[30075]: pgmap v1616: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:32.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:53:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:53:32.734 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:53:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:53:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:33 smithi138 ceph-mon[33706]: pgmap v1617: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:33.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:33 smithi088 ceph-mon[39202]: pgmap v1617: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:33.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:33 smithi088 ceph-mon[30075]: pgmap v1617: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:33.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:35.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:35 smithi138 ceph-mon[33706]: pgmap v1618: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:35 smithi088 ceph-mon[39202]: pgmap v1618: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:35.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:35 smithi088 ceph-mon[30075]: pgmap v1618: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:37.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:37 smithi138 ceph-mon[33706]: pgmap v1619: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:37.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:37 smithi088 ceph-mon[30075]: pgmap v1619: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:37.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:37 smithi088 ceph-mon[39202]: pgmap v1619: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:39.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:39 smithi138 ceph-mon[33706]: pgmap v1620: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:39.484 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:39 smithi088 ceph-mon[30075]: pgmap v1620: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:39.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:39 smithi088 ceph-mon[39202]: pgmap v1620: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:41.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:41 smithi138 ceph-mon[33706]: pgmap v1621: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:41 smithi088 ceph-mon[39202]: pgmap v1621: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:41.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:41 smithi088 ceph-mon[30075]: pgmap v1621: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:42.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:42.468 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:42.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:42.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:53:42 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:53:42.735 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:53:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:53:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:43.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:43 smithi138 ceph-mon[33706]: pgmap v1622: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:43.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:43 smithi088 ceph-mon[39202]: pgmap v1622: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:43.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:43 smithi088 ceph-mon[30075]: pgmap v1622: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:43.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:44.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:44 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:53:44.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:44 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:53:44.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:44 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:53:44.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:44 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:53:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:53:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:53:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:53:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:53:44.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:53:44.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:53:44.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:53:44.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:44 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:53:45.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:45 smithi138 ceph-mon[33706]: pgmap v1623: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:45.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:45 smithi088 ceph-mon[30075]: pgmap v1623: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:45.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:45 smithi088 ceph-mon[39202]: pgmap v1623: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:47.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:47 smithi138 ceph-mon[33706]: pgmap v1624: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:47 smithi088 ceph-mon[39202]: pgmap v1624: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:47.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:47 smithi088 ceph-mon[30075]: pgmap v1624: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:49 smithi088 ceph-mon[39202]: pgmap v1625: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:49.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:49 smithi088 ceph-mon[30075]: pgmap v1625: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:49.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:49 smithi138 ceph-mon[33706]: pgmap v1625: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:51.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:51 smithi088 ceph-mon[30075]: pgmap v1626: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:51 smithi088 ceph-mon[39202]: pgmap v1626: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:51.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:51 smithi138 ceph-mon[33706]: pgmap v1626: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:52.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:53:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:53:52.735 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:53:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:53:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:53 smithi088 ceph-mon[39202]: pgmap v1627: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:53 smithi088 ceph-mon[30075]: pgmap v1627: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:53.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:53.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:53 smithi138 ceph-mon[33706]: pgmap v1627: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:53.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:55.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:55 smithi138 ceph-mon[33706]: pgmap v1628: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:55.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:55 smithi088 ceph-mon[39202]: pgmap v1628: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:55.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:55 smithi088 ceph-mon[30075]: pgmap v1628: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:57.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:57 smithi138 ceph-mon[33706]: pgmap v1629: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:57.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:57.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:57 smithi088 ceph-mon[39202]: pgmap v1629: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:57.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:57.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:57 smithi088 ceph-mon[30075]: pgmap v1629: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:57.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:59.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:53:59 smithi138 ceph-mon[33706]: pgmap v1630: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:59.734 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:53:59 smithi088 ceph-mon[30075]: pgmap v1630: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:53:59.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:53:59 smithi088 ceph-mon[39202]: pgmap v1630: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:01.528 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:01 smithi138 ceph-mon[33706]: pgmap v1631: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:01.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:01 smithi088 ceph-mon[30075]: pgmap v1631: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:01.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:01 smithi088 ceph-mon[39202]: pgmap v1631: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:02.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:54:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:54:02.735 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:54:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:54:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:54:03.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:03 smithi138 ceph-mon[33706]: pgmap v1632: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:03.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:03.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:03 smithi088 ceph-mon[30075]: pgmap v1632: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:03.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:03.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:03 smithi088 ceph-mon[39202]: pgmap v1632: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:03.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:05.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:05 smithi138 ceph-mon[33706]: pgmap v1633: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:05.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:05 smithi088 ceph-mon[30075]: pgmap v1633: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:05.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:05 smithi088 ceph-mon[39202]: pgmap v1633: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:07.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:07 smithi138 ceph-mon[33706]: pgmap v1634: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:07.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:07 smithi088 ceph-mon[30075]: pgmap v1634: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:07.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:07 smithi088 ceph-mon[39202]: pgmap v1634: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:09.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:09 smithi138 ceph-mon[33706]: pgmap v1635: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:09.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:09 smithi088 ceph-mon[30075]: pgmap v1635: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:09.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:09 smithi088 ceph-mon[39202]: pgmap v1635: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:11.537 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:11 smithi088 ceph-mon[30075]: pgmap v1636: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:11.538 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:11 smithi088 ceph-mon[39202]: pgmap v1636: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:11.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:11 smithi138 ceph-mon[33706]: pgmap v1636: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:12.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:12.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:54:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:54:12.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:12.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:12.735 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:54:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:54:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:54:13.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:13 smithi138 ceph-mon[33706]: pgmap v1637: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:13.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:13.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:13 smithi088 ceph-mon[39202]: pgmap v1637: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:13.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:13.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:13 smithi088 ceph-mon[30075]: pgmap v1637: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:13.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:15.725 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:15 smithi138 ceph-mon[33706]: pgmap v1638: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:15.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:15 smithi088 ceph-mon[30075]: pgmap v1638: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:15.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:15 smithi088 ceph-mon[39202]: pgmap v1638: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:17.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:17 smithi138 ceph-mon[33706]: pgmap v1639: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:17.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:17 smithi088 ceph-mon[39202]: pgmap v1639: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:17.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:17 smithi088 ceph-mon[30075]: pgmap v1639: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:19.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:19 smithi138 ceph-mon[33706]: pgmap v1640: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:19.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:19 smithi088 ceph-mon[30075]: pgmap v1640: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:19.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:19 smithi088 ceph-mon[39202]: pgmap v1640: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:54:21.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:21 smithi138 ceph-mon[33706]: pgmap v1641: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:21.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:21 smithi088 ceph-mon[30075]: pgmap v1641: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:21.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:21 smithi088 ceph-mon[39202]: pgmap v1641: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:54:22.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:54:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:54:22.735 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:54:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:54:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:54:23.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:23 smithi138 ceph-mon[33706]: pgmap v1642: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:23.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:23.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:23 smithi088 ceph-mon[30075]: pgmap v1642: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:23.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:23.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:23 smithi088 ceph-mon[39202]: pgmap v1642: 260 pgs: 260 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:23.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:24.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:24 smithi138 ceph-mon[33706]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T21:54:24.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:24 smithi088 ceph-mon[30075]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T21:54:24.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:24 smithi088 ceph-mon[39202]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T21:54:25.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:25 smithi138 ceph-mon[33706]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:54:25.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:25 smithi138 ceph-mon[33706]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:25.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:25 smithi138 ceph-mon[33706]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T21:54:25.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:25 smithi088 ceph-mon[30075]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:54:25.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:25 smithi088 ceph-mon[30075]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:25.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:25 smithi088 ceph-mon[30075]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T21:54:25.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:25 smithi088 ceph-mon[39202]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:54:25.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:25 smithi088 ceph-mon[39202]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:25.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:25 smithi088 ceph-mon[39202]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T21:54:26.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:26 smithi138 ceph-mon[33706]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T21:54:26.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:26 smithi088 ceph-mon[39202]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T21:54:26.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:26 smithi088 ceph-mon[30075]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T21:54:27.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:27 smithi138 ceph-mon[33706]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T21:54:27.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:27 smithi138 ceph-mon[33706]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T21:54:27.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:27.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:27 smithi088 ceph-mon[30075]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T21:54:27.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:27 smithi088 ceph-mon[30075]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T21:54:27.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:27.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:27 smithi088 ceph-mon[39202]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T21:54:27.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:27 smithi088 ceph-mon[39202]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T21:54:27.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:28.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:28 smithi138 ceph-mon[33706]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T21:54:28.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:28 smithi138 ceph-mon[33706]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T21:54:28.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:28 smithi088 ceph-mon[30075]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T21:54:28.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:28 smithi088 ceph-mon[30075]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T21:54:28.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:28 smithi088 ceph-mon[39202]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T21:54:28.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:28 smithi088 ceph-mon[39202]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T21:54:29.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:29 smithi138 ceph-mon[33706]: pgmap v1650: 260 pgs: 64 unknown, 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:29.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:29 smithi138 ceph-mon[33706]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T21:54:29.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:29 smithi088 ceph-mon[30075]: pgmap v1650: 260 pgs: 64 unknown, 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:29.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:29 smithi088 ceph-mon[30075]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T21:54:29.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:29 smithi088 ceph-mon[39202]: pgmap v1650: 260 pgs: 64 unknown, 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:29.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:29 smithi088 ceph-mon[39202]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T21:54:30.346 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: handle_notify cookie 94785921205984 notify_id 3195455668233 notifier_gid 34928 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (1800749 ms) 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: Trying... 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: handle_notify cookie 94785933078544 notify_id 3208340570112 notifier_gid 37530 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: Waiting for 3.000000000s 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: Timed out. 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: Flushing... 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: Flushed... 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7043 ms) 2024-03-26T21:54:30.347 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (1807792 ms total) 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (1807792 ms total) 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79441 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79441 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79921 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79921 2024-03-26T21:54:30.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80343 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80343 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80037 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80037 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80439 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80439 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79850 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79850 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79305 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79305 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80525 2024-03-26T21:54:30.349 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80525 2024-03-26T21:54:31.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:31 smithi138 ceph-mon[33706]: pgmap v1653: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:54:31.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:31 smithi138 ceph-mon[33706]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T21:54:31.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:31 smithi138 ceph-mon[33706]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:31.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:31 smithi088 ceph-mon[30075]: pgmap v1653: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:54:31.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:31 smithi088 ceph-mon[30075]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T21:54:31.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:31 smithi088 ceph-mon[30075]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:31.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:31 smithi088 ceph-mon[39202]: pgmap v1653: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:54:31.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:31 smithi088 ceph-mon[39202]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T21:54:31.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:31 smithi088 ceph-mon[39202]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:32.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:54:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:54:32.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:32 smithi138 ceph-mon[33706]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T21:54:32.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:32 smithi138 ceph-mon[33706]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T21:54:32.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:32 smithi088 ceph-mon[39202]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T21:54:32.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:32 smithi088 ceph-mon[39202]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T21:54:32.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:32 smithi088 ceph-mon[30075]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T21:54:32.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:32 smithi088 ceph-mon[30075]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T21:54:32.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:54:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:54:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:54:33.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:33 smithi138 ceph-mon[33706]: pgmap v1656: 228 pgs: 32 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:54:33.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:33.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:33 smithi138 ceph-mon[33706]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T21:54:33.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:33 smithi088 ceph-mon[39202]: pgmap v1656: 228 pgs: 32 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:54:33.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:33.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:33 smithi088 ceph-mon[39202]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T21:54:33.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:33 smithi088 ceph-mon[30075]: pgmap v1656: 228 pgs: 32 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:54:33.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:33.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:33 smithi088 ceph-mon[30075]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T21:54:35.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:35 smithi138 ceph-mon[33706]: pgmap v1659: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:54:35.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:35 smithi138 ceph-mon[33706]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T21:54:35.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:35 smithi088 ceph-mon[39202]: pgmap v1659: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:54:35.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:35 smithi088 ceph-mon[39202]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T21:54:35.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:35 smithi088 ceph-mon[30075]: pgmap v1659: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:54:35.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:35 smithi088 ceph-mon[30075]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T21:54:36.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:36 smithi138 ceph-mon[33706]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T21:54:36.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:36 smithi088 ceph-mon[39202]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T21:54:36.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:36 smithi088 ceph-mon[30075]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T21:54:37.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:37 smithi138 ceph-mon[33706]: pgmap v1662: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:37.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:37 smithi138 ceph-mon[33706]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:37.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:37 smithi138 ceph-mon[33706]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T21:54:37.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:37 smithi088 ceph-mon[30075]: pgmap v1662: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:37.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:37 smithi088 ceph-mon[30075]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:37.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:37 smithi088 ceph-mon[30075]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T21:54:37.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:37 smithi088 ceph-mon[39202]: pgmap v1662: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:37.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:37 smithi088 ceph-mon[39202]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:37.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:37 smithi088 ceph-mon[39202]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T21:54:38.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:38 smithi138 ceph-mon[33706]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T21:54:38.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:38 smithi088 ceph-mon[30075]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T21:54:38.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:38 smithi088 ceph-mon[39202]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T21:54:39.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:39 smithi138 ceph-mon[33706]: pgmap v1665: 228 pgs: 32 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:39.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:39 smithi138 ceph-mon[33706]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T21:54:39.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:39 smithi088 ceph-mon[30075]: pgmap v1665: 228 pgs: 32 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:39.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:39 smithi088 ceph-mon[30075]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T21:54:39.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:39 smithi088 ceph-mon[39202]: pgmap v1665: 228 pgs: 32 unknown, 32 creating+peering, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:39.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:39 smithi088 ceph-mon[39202]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T21:54:40.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:40 smithi138 ceph-mon[33706]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T21:54:40.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:40 smithi088 ceph-mon[39202]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T21:54:40.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:40 smithi088 ceph-mon[30075]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T21:54:41.412 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: Running main() from gmock_main.cc 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [----------] Global test environment set-up. 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (1800757 ms) 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3023 ms) 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3004 ms) 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3008 ms) 2024-03-26T21:54:41.413 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3033 ms) 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3027 ms) 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (3011 ms) 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (1818863 ms total) 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [----------] Global test environment tear-down 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (1818863 ms total) 2024-03-26T21:54:41.414 INFO:tasks.workunit.client.0.smithi088.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-26T21:54:41.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:41 smithi138 ceph-mon[33706]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:54:41.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:41 smithi138 ceph-mon[33706]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T21:54:41.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:41 smithi088 ceph-mon[30075]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:54:41.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:41 smithi088 ceph-mon[30075]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T21:54:41.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:41 smithi088 ceph-mon[39202]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:54:41.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:41 smithi088 ceph-mon[39202]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T21:54:42.425 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:54:42 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:54:42.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:42 smithi138 ceph-mon[33706]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T21:54:42.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:42.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:42 smithi138 ceph-mon[33706]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:42.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:42 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:42.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[30075]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[30075]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[39202]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[39202]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:42.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:54:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:54:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:54:43.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:43 smithi138 ceph-mon[33706]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T21:54:43.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:43 smithi088 ceph-mon[30075]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T21:54:43.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:43 smithi088 ceph-mon[39202]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T21:54:44.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:44 smithi138 ceph-mon[33706]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T21:54:44.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:44 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:44.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:44 smithi138 ceph-mon[33706]: pgmap v1674: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:44.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:44 smithi088 ceph-mon[30075]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T21:54:44.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:44 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:44.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:44 smithi088 ceph-mon[30075]: pgmap v1674: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:44.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:44 smithi088 ceph-mon[39202]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T21:54:44.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:44 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:44.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:44 smithi088 ceph-mon[39202]: pgmap v1674: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:45.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:45 smithi138 ceph-mon[33706]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T21:54:45.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:45 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:45.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:45 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:45.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:45 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:54:45.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[30075]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T21:54:45.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:45.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:45.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:54:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[39202]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T21:54:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:45 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:54:46.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:46 smithi138 ceph-mon[33706]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T21:54:46.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:46 smithi138 ceph-mon[33706]: pgmap v1677: 196 pgs: 196 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:54:46.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:46 smithi088 ceph-mon[30075]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T21:54:46.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:46 smithi088 ceph-mon[30075]: pgmap v1677: 196 pgs: 196 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:54:46.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:46 smithi088 ceph-mon[39202]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T21:54:46.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:46 smithi088 ceph-mon[39202]: pgmap v1677: 196 pgs: 196 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T21:54:47.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:47 smithi138 ceph-mon[33706]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T21:54:47.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:47 smithi088 ceph-mon[30075]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T21:54:47.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:47 smithi088 ceph-mon[39202]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T21:54:48.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:48 smithi138 ceph-mon[33706]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T21:54:48.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:48 smithi138 ceph-mon[33706]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:48.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:48 smithi088 ceph-mon[30075]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T21:54:48.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:48 smithi088 ceph-mon[30075]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:48 smithi088 ceph-mon[39202]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T21:54:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:48 smithi088 ceph-mon[39202]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:49.735 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:49 smithi088 ceph-mon[30075]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T21:54:49.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:49 smithi088 ceph-mon[39202]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T21:54:49.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:49 smithi138 ceph-mon[33706]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T21:54:50.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:50 smithi138 ceph-mon[33706]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T21:54:50.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:50 smithi138 ceph-mon[33706]: pgmap v1683: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:54:50.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:50 smithi088 ceph-mon[39202]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T21:54:50.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:50 smithi088 ceph-mon[39202]: pgmap v1683: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:54:50.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:50 smithi088 ceph-mon[30075]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T21:54:50.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:50 smithi088 ceph-mon[30075]: pgmap v1683: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:54:51.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:51 smithi138 ceph-mon[33706]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T21:54:51.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:51 smithi088 ceph-mon[30075]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T21:54:51.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:51 smithi088 ceph-mon[39202]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T21:54:52.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:54:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:54:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:52 smithi088 ceph-mon[39202]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T21:54:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:52 smithi088 ceph-mon[39202]: pgmap v1686: 164 pgs: 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:52 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:52.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:52 smithi088 ceph-mon[30075]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T21:54:52.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:52 smithi088 ceph-mon[30075]: pgmap v1686: 164 pgs: 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:52.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:52 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:52.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:54:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:54:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:54:52.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:52 smithi138 ceph-mon[33706]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T21:54:52.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:52 smithi138 ceph-mon[33706]: pgmap v1686: 164 pgs: 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:52.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:52 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:53.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:53 smithi138 ceph-mon[33706]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:53.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:53 smithi138 ceph-mon[33706]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T21:54:53.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:53 smithi088 ceph-mon[39202]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:53.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:53 smithi088 ceph-mon[39202]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T21:54:53.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:53 smithi088 ceph-mon[30075]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:53.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:53 smithi088 ceph-mon[30075]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T21:54:54.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:54 smithi138 ceph-mon[33706]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T21:54:54.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:54 smithi138 ceph-mon[33706]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:54.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:54 smithi088 ceph-mon[30075]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T21:54:54.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:54 smithi088 ceph-mon[30075]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:54 smithi088 ceph-mon[39202]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T21:54:54.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:54 smithi088 ceph-mon[39202]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:55.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:55 smithi138 ceph-mon[33706]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T21:54:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:55 smithi088 ceph-mon[39202]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T21:54:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:55 smithi088 ceph-mon[30075]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T21:54:56.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:56 smithi138 ceph-mon[33706]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T21:54:56.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:56 smithi138 ceph-mon[33706]: pgmap v1692: 196 pgs: 196 active+clean; 583 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:54:56.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:56 smithi138 ceph-mon[33706]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T21:54:56.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:56 smithi088 ceph-mon[30075]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T21:54:56.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:56 smithi088 ceph-mon[30075]: pgmap v1692: 196 pgs: 196 active+clean; 583 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:54:56.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:56 smithi088 ceph-mon[30075]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T21:54:56.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:56 smithi088 ceph-mon[39202]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T21:54:56.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:56 smithi088 ceph-mon[39202]: pgmap v1692: 196 pgs: 196 active+clean; 583 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T21:54:56.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:56 smithi088 ceph-mon[39202]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T21:54:57.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:57.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:57 smithi138 ceph-mon[33706]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T21:54:57.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:57.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:57 smithi088 ceph-mon[39202]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T21:54:57.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:57.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:57 smithi088 ceph-mon[30075]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T21:54:58.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:58 smithi138 ceph-mon[33706]: pgmap v1695: 164 pgs: 164 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:58.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:58 smithi088 ceph-mon[39202]: pgmap v1695: 164 pgs: 164 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:58.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:58 smithi088 ceph-mon[30075]: pgmap v1695: 164 pgs: 164 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:59.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:59 smithi138 ceph-mon[33706]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:59.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:54:59 smithi138 ceph-mon[33706]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T21:54:59.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:59 smithi088 ceph-mon[39202]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:59.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:54:59 smithi088 ceph-mon[39202]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T21:54:59.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:59 smithi088 ceph-mon[30075]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:54:59.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:54:59 smithi088 ceph-mon[30075]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T21:55:00.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:00 smithi138 ceph-mon[33706]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T21:55:00.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:00 smithi138 ceph-mon[33706]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:00.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:00 smithi088 ceph-mon[39202]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T21:55:00.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:00 smithi088 ceph-mon[39202]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:00.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:00 smithi088 ceph-mon[30075]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T21:55:00.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:00 smithi088 ceph-mon[30075]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:01.878 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:01 smithi138 ceph-mon[33706]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T21:55:01.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:01 smithi088 ceph-mon[39202]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T21:55:01.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:01 smithi088 ceph-mon[30075]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T21:55:02.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:55:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:55:02.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:02 smithi088 ceph-mon[30075]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T21:55:02.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:02 smithi088 ceph-mon[30075]: pgmap v1701: 196 pgs: 196 active+clean; 584 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:55:02.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:02 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:02.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:55:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:55:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:55:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:02 smithi088 ceph-mon[39202]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T21:55:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:02 smithi088 ceph-mon[39202]: pgmap v1701: 196 pgs: 196 active+clean; 584 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:55:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:02 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:02.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:02 smithi138 ceph-mon[33706]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T21:55:02.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:02 smithi138 ceph-mon[33706]: pgmap v1701: 196 pgs: 196 active+clean; 584 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:55:02.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:02 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:03.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:03 smithi138 ceph-mon[33706]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T21:55:03.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:03 smithi088 ceph-mon[39202]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T21:55:03.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:03 smithi088 ceph-mon[30075]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T21:55:04.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:04 smithi138 ceph-mon[33706]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T21:55:04.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:04 smithi138 ceph-mon[33706]: pgmap v1704: 196 pgs: 196 active+clean; 584 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:55:04.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:04 smithi138 ceph-mon[33706]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T21:55:04.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:04 smithi088 ceph-mon[39202]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T21:55:04.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:04 smithi088 ceph-mon[39202]: pgmap v1704: 196 pgs: 196 active+clean; 584 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:55:04.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:04 smithi088 ceph-mon[39202]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T21:55:04.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:04 smithi088 ceph-mon[30075]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T21:55:04.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:04 smithi088 ceph-mon[30075]: pgmap v1704: 196 pgs: 196 active+clean; 584 KiB data, 833 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T21:55:04.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:04 smithi088 ceph-mon[30075]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T21:55:06.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:06 smithi138 ceph-mon[33706]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T21:55:06.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:06 smithi138 ceph-mon[33706]: pgmap v1707: 164 pgs: 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:55:06.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:06 smithi088 ceph-mon[30075]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T21:55:06.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:06 smithi088 ceph-mon[30075]: pgmap v1707: 164 pgs: 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:55:06.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:06 smithi088 ceph-mon[39202]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T21:55:06.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:06 smithi088 ceph-mon[39202]: pgmap v1707: 164 pgs: 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:55:07.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:07 smithi138 ceph-mon[33706]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:07.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:07 smithi138 ceph-mon[33706]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T21:55:07.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:07 smithi088 ceph-mon[30075]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:07.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:07 smithi088 ceph-mon[30075]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T21:55:07.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:07 smithi088 ceph-mon[39202]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:07.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:07 smithi088 ceph-mon[39202]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T21:55:08.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:08 smithi138 ceph-mon[33706]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T21:55:08.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:08 smithi138 ceph-mon[33706]: pgmap v1710: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:55:08.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:08 smithi088 ceph-mon[39202]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T21:55:08.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:08 smithi088 ceph-mon[39202]: pgmap v1710: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:55:08.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:08 smithi088 ceph-mon[30075]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T21:55:08.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:08 smithi088 ceph-mon[30075]: pgmap v1710: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T21:55:09.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:09 smithi138 ceph-mon[33706]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T21:55:09.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:09 smithi088 ceph-mon[39202]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T21:55:09.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:09 smithi088 ceph-mon[30075]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T21:55:10.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:10 smithi138 ceph-mon[33706]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T21:55:10.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:10 smithi138 ceph-mon[33706]: pgmap v1713: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:10.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:10 smithi088 ceph-mon[39202]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T21:55:10.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:10 smithi088 ceph-mon[39202]: pgmap v1713: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:10.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:10 smithi088 ceph-mon[30075]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T21:55:10.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:10 smithi088 ceph-mon[30075]: pgmap v1713: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 850 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:11.887 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[39202]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T21:55:11.887 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]: dispatch 2024-03-26T21:55:11.887 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]: dispatch 2024-03-26T21:55:11.887 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]: dispatch 2024-03-26T21:55:11.887 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]: dispatch 2024-03-26T21:55:11.887 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[30075]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T21:55:11.887 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]: dispatch 2024-03-26T21:55:11.888 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]: dispatch 2024-03-26T21:55:11.888 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]: dispatch 2024-03-26T21:55:11.888 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:11 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]: dispatch 2024-03-26T21:55:11.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:11 smithi138 ceph-mon[33706]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T21:55:11.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]: dispatch 2024-03-26T21:55:11.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:11 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]: dispatch 2024-03-26T21:55:11.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]: dispatch 2024-03-26T21:55:11.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:11 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]: dispatch 2024-03-26T21:55:12.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:55:12.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:55:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:55:12.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]': finished 2024-03-26T21:55:12.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]': finished 2024-03-26T21:55:12.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[39202]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T21:55:12.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:12.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[39202]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:12.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[39202]: pgmap v1716: 164 pgs: 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]': finished 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]': finished 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[30075]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[30075]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[30075]: pgmap v1716: 164 pgs: 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:12 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:12.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.5", "id": [5, 2]}]': finished 2024-03-26T21:55:12.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "343.10", "id": [5, 6]}]': finished 2024-03-26T21:55:12.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-mon[33706]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T21:55:12.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:12.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-mon[33706]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:12.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-mon[33706]: pgmap v1716: 164 pgs: 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:12.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:12 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:13.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:13 smithi138 ceph-mon[33706]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T21:55:13.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:13 smithi088 ceph-mon[39202]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T21:55:13.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:13 smithi088 ceph-mon[30075]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T21:55:14.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:14 smithi138 ceph-mon[33706]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T21:55:14.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:14 smithi138 ceph-mon[33706]: pgmap v1719: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:14 smithi088 ceph-mon[39202]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T21:55:14.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:14 smithi088 ceph-mon[39202]: pgmap v1719: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:14 smithi088 ceph-mon[30075]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T21:55:14.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:14 smithi088 ceph-mon[30075]: pgmap v1719: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:15.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:15 smithi138 ceph-mon[33706]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T21:55:15.985 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:15 smithi088 ceph-mon[30075]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T21:55:15.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:15 smithi088 ceph-mon[39202]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T21:55:16.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:16 smithi138 ceph-mon[33706]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T21:55:16.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:16 smithi138 ceph-mon[33706]: pgmap v1722: 164 pgs: 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:16.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:16 smithi088 ceph-mon[39202]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T21:55:16.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:16 smithi088 ceph-mon[39202]: pgmap v1722: 164 pgs: 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:16.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:16 smithi088 ceph-mon[30075]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T21:55:16.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:16 smithi088 ceph-mon[30075]: pgmap v1722: 164 pgs: 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:17.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:17 smithi138 ceph-mon[33706]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T21:55:17.977 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:17 smithi138 ceph-mon[33706]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:17.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:17 smithi088 ceph-mon[39202]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T21:55:17.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:17 smithi088 ceph-mon[39202]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:17.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:17 smithi088 ceph-mon[30075]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T21:55:17.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:17 smithi088 ceph-mon[30075]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:18.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:18 smithi138 ceph-mon[33706]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T21:55:18.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:18 smithi138 ceph-mon[33706]: pgmap v1725: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:18.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:18 smithi088 ceph-mon[39202]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T21:55:18.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:18 smithi088 ceph-mon[39202]: pgmap v1725: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:18.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:18 smithi088 ceph-mon[30075]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T21:55:18.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:18 smithi088 ceph-mon[30075]: pgmap v1725: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:55:19.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:19 smithi138 ceph-mon[33706]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T21:55:19.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:19 smithi088 ceph-mon[39202]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T21:55:19.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:19 smithi088 ceph-mon[30075]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T21:55:20.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:20 smithi138 ceph-mon[33706]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T21:55:20.976 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:20 smithi138 ceph-mon[33706]: pgmap v1728: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:20.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:20 smithi088 ceph-mon[39202]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T21:55:20.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:20 smithi088 ceph-mon[39202]: pgmap v1728: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:20.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:20 smithi088 ceph-mon[30075]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T21:55:20.986 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:20 smithi088 ceph-mon[30075]: pgmap v1728: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:22.469 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:22 smithi088 ceph-mon[30075]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:22.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:22 smithi088 ceph-mon[39202]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:22.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:22 smithi138 ceph-mon[33706]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:22.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:55:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:55:22.735 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:55:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:55:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:55:23.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:23 smithi138 ceph-mon[33706]: pgmap v1729: 196 pgs: 196 active+clean; 583 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 939 B/s rd, 187 B/s wr, 1 op/s 2024-03-26T21:55:23.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:23.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:23 smithi088 ceph-mon[30075]: pgmap v1729: 196 pgs: 196 active+clean; 583 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 939 B/s rd, 187 B/s wr, 1 op/s 2024-03-26T21:55:23.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:23.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:23 smithi088 ceph-mon[39202]: pgmap v1729: 196 pgs: 196 active+clean; 583 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 939 B/s rd, 187 B/s wr, 1 op/s 2024-03-26T21:55:23.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:25.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:25 smithi138 ceph-mon[33706]: pgmap v1730: 196 pgs: 196 active+clean; 583 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 158 B/s wr, 0 op/s 2024-03-26T21:55:25.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:25 smithi088 ceph-mon[39202]: pgmap v1730: 196 pgs: 196 active+clean; 583 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 158 B/s wr, 0 op/s 2024-03-26T21:55:25.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:25 smithi088 ceph-mon[30075]: pgmap v1730: 196 pgs: 196 active+clean; 583 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 158 B/s wr, 0 op/s 2024-03-26T21:55:27.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:27 smithi138 ceph-mon[33706]: pgmap v1731: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-26T21:55:27.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:27.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:27 smithi138 ceph-mon[33706]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T21:55:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:27 smithi088 ceph-mon[30075]: pgmap v1731: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-26T21:55:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:27.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:27 smithi088 ceph-mon[30075]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T21:55:27.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:27 smithi088 ceph-mon[39202]: pgmap v1731: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-26T21:55:27.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:27.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:27 smithi088 ceph-mon[39202]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T21:55:29.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:29 smithi138 ceph-mon[33706]: pgmap v1733: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T21:55:29.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:29 smithi088 ceph-mon[39202]: pgmap v1733: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T21:55:29.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:29 smithi088 ceph-mon[30075]: pgmap v1733: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T21:55:31.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:31 smithi138 ceph-mon[33706]: pgmap v1734: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T21:55:31.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:31 smithi088 ceph-mon[39202]: pgmap v1734: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T21:55:31.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:31 smithi088 ceph-mon[30075]: pgmap v1734: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T21:55:32.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:55:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:55:32.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:55:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:55:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:55:33.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:33 smithi138 ceph-mon[33706]: pgmap v1735: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:33.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:33.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:33 smithi088 ceph-mon[39202]: pgmap v1735: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:33.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:33 smithi088 ceph-mon[30075]: pgmap v1735: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:33.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: Running main() from gmock_main.cc 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [----------] Global test environment set-up. 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (1802773 ms) 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5024 ms) 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4025 ms) 2024-03-26T21:55:35.208 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5044 ms) 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7061 ms) 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (1823927 ms total) 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5034 ms) 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6033 ms) 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8056 ms) 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6047 ms) 2024-03-26T21:55:35.209 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4040 ms) 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi088-80614-11. 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: Waiting for snaps to purge. 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19530 ms) 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48740 ms total) 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [----------] Global test environment tear-down 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (1872667 ms total) 2024-03-26T21:55:35.210 INFO:tasks.workunit.client.0.smithi088.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-26T21:55:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79735 2024-03-26T21:55:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79735 2024-03-26T21:55:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79147 2024-03-26T21:55:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79147 2024-03-26T21:55:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79205 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79205 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79598 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79598 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79245 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79245 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79774 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79774 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=79825 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 79825 2024-03-26T21:55:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80239 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80239 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=80583 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 80583 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ exit 0 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ cleanup 2024-03-26T21:55:35.213 INFO:tasks.workunit.client.0.smithi088.stderr:+ pkill -P 79141 2024-03-26T21:55:35.223 INFO:tasks.workunit.client.0.smithi088.stderr:+ true 2024-03-26T21:55:35.224 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T21:55:35.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T21:55:35.256 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-26T21:55:35.257 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-26T21:55:35.258 DEBUG:teuthology.orchestra.run.smithi088:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-26T21:55:35.317 INFO:tasks.workunit.client.0.smithi088.stderr:++ uuidgen 2024-03-26T21:55:35.322 INFO:tasks.workunit.client.0.smithi088.stderr:+ p=3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:55:35.322 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool create 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 12 2024-03-26T21:55:35.410 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.412+0000 7fdc2bebb640 1 -- 172.21.15.88:0/2292791745 >> v1:172.21.15.88:6789/0 conn(0x7fdc2410d310 legacy=0x7fdc2410f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:35.410 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.412+0000 7fdc2bebb640 1 -- 172.21.15.88:0/2292791745 shutdown_connections 2024-03-26T21:55:35.410 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.412+0000 7fdc2bebb640 1 -- 172.21.15.88:0/2292791745 >> 172.21.15.88:0/2292791745 conn(0x7fdc24077560 msgr2=0x7fdc240fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:35.410 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.412+0000 7fdc2bebb640 1 -- 172.21.15.88:0/2292791745 shutdown_connections 2024-03-26T21:55:35.410 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.412+0000 7fdc2bebb640 1 -- 172.21.15.88:0/2292791745 wait complete. 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc2bebb640 1 Processor -- start 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc2bebb640 1 -- start start 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc2bebb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc241a57b0 con 0x7fdc2410d310 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc2bebb640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc241a69b0 con 0x7fdc24109740 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc2bebb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc241a7bf0 con 0x7fdc241068a0 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc29c57640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fdc241068a0 0x7fdc2410c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53632/0 (socket says 172.21.15.88:53632) 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc29c57640 1 -- 172.21.15.88:0/4118015100 learned_addr learned my addr 172.21.15.88:0/4118015100 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523872327 0 0) 0x7fdc241a57b0 con 0x7fdc2410d310 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc10003520 con 0x7fdc2410d310 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114669924 0 0) 0x7fdc241a7bf0 con 0x7fdc241068a0 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc241a57b0 con 0x7fdc241068a0 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2268186811 0 0) 0x7fdc10003520 con 0x7fdc2410d310 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc241a7bf0 con 0x7fdc2410d310 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3491900163 0 0) 0x7fdc241a57b0 con 0x7fdc241068a0 2024-03-26T21:55:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc10003520 con 0x7fdc241068a0 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fdc1c0016f0 con 0x7fdc2410d310 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fdc200019c0 con 0x7fdc241068a0 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.413+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179702714 0 0) 0x7fdc241a69b0 con 0x7fdc24109740 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc241a57b0 con 0x7fdc24109740 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1555869053 0 0) 0x7fdc241a7bf0 con 0x7fdc2410d310 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 >> v1:172.21.15.88:6790/0 conn(0x7fdc241068a0 legacy=0x7fdc2410c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 >> v1:172.21.15.138:6789/0 conn(0x7fdc24109740 legacy=0x7fdc241a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc241a8e30 con 0x7fdc2410d310 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc241a59e0 con 0x7fdc2410d310 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc241a5eb0 con 0x7fdc2410d310 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fdc1c001980 con 0x7fdc2410d310 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.414+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fdc1c004e30 con 0x7fdc2410d310 2024-03-26T21:55:35.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.415+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc08005080 con 0x7fdc2410d310 2024-03-26T21:55:35.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.415+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fdc1c001e70 con 0x7fdc2410d310 2024-03-26T21:55:35.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.416+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(803..803 src has 268..803) ==== 7411+0+0 (unknown 3245279464 0 0) 0x7fdc1c0894b0 con 0x7fdc2410d310 2024-03-26T21:55:35.416 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.420+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fdc1c056400 con 0x7fdc2410d310 2024-03-26T21:55:35.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:35 smithi138 ceph-mon[33706]: pgmap v1736: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:35 smithi088 ceph-mon[39202]: pgmap v1736: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:35.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:35 smithi088 ceph-mon[30075]: pgmap v1736: 196 pgs: 196 active+clean; 583 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:35.596 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:35.601+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12} v 0) -- 0x7fdc08005370 con 0x7fdc2410d310 2024-03-26T21:55:36.215 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.220+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]=0 pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' created v804) ==== 176+0+0 (unknown 3746613780 0 0) 0x7fdc1c05a0b0 con 0x7fdc2410d310 2024-03-26T21:55:36.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.336+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12} v 0) -- 0x7fdc08001170 con 0x7fdc2410d310 2024-03-26T21:55:36.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.336+0000 7fdc1affd640 1 -- 172.21.15.88:0/4118015100 <== mon.0 v1:172.21.15.88:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]=0 pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' already exists v804) ==== 183+0+0 (unknown 1735722621 0 0) 0x7fdc08001170 con 0x7fdc2410d310 2024-03-26T21:55:36.332 INFO:tasks.workunit.client.0.smithi088.stderr:pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' already exists 2024-03-26T21:55:36.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.338+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 >> v1:172.21.15.88:6800/3939589528 conn(0x7fdc1006f550 legacy=0x7fdc10071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:36.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.338+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 >> v1:172.21.15.88:6789/0 conn(0x7fdc2410d310 legacy=0x7fdc241a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:36.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.338+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 shutdown_connections 2024-03-26T21:55:36.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.338+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 >> 172.21.15.88:0/4118015100 conn(0x7fdc24077560 msgr2=0x7fdc2410d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:36.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.338+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 shutdown_connections 2024-03-26T21:55:36.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.338+0000 7fdc2bebb640 1 -- 172.21.15.88:0/4118015100 wait complete. 2024-03-26T21:55:36.343 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 max_objects 10 2024-03-26T21:55:36.430 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.431+0000 7f1af5439640 1 -- 172.21.15.88:0/238030748 >> v1:172.21.15.88:6790/0 conn(0x7f1af010d300 legacy=0x7f1af010f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:36.430 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.431+0000 7f1af5439640 1 -- 172.21.15.88:0/238030748 shutdown_connections 2024-03-26T21:55:36.430 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.431+0000 7f1af5439640 1 -- 172.21.15.88:0/238030748 >> 172.21.15.88:0/238030748 conn(0x7f1af00fc350 msgr2=0x7f1af00fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:36.430 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.431+0000 7f1af5439640 1 -- 172.21.15.88:0/238030748 shutdown_connections 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.431+0000 7f1af5439640 1 -- 172.21.15.88:0/238030748 wait complete. 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1af5439640 1 Processor -- start 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1af5439640 1 -- start start 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1af5439640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1af01a5980 con 0x7f1af0109730 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1af5439640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1af01a6bc0 con 0x7f1af010d300 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1af5439640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1af01a7e00 con 0x7f1af0106890 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1aeeffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f1af0106890 0x7f1af010cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53644/0 (socket says 172.21.15.88:53644) 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1aeeffd640 1 -- 172.21.15.88:0/507250870 learned_addr learned my addr 172.21.15.88:0/507250870 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1ae7fff640 1 --1- 172.21.15.88:0/507250870 >> v1:172.21.15.88:6789/0 conn(0x7f1af0109730 0x7f1af01a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52632/0 (socket says 172.21.15.88:52632) 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2691740135 0 0) 0x7f1af01a7e00 con 0x7f1af0106890 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1adc003160 con 0x7f1af0106890 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.432+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469169597 0 0) 0x7f1af01a5980 con 0x7f1af0109730 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1af01a7e00 con 0x7f1af0109730 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4114841244 0 0) 0x7f1adc003160 con 0x7f1af0106890 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1af01a5980 con 0x7f1af0106890 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f1ae0001a30 con 0x7f1af0106890 2024-03-26T21:55:36.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2772098331 0 0) 0x7f1af01a6bc0 con 0x7f1af010d300 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1adc003160 con 0x7f1af010d300 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3594026935 0 0) 0x7f1af01a7e00 con 0x7f1af0109730 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1af01a6bc0 con 0x7f1af0109730 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f1ad80017b0 con 0x7f1af0109730 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3593039409 0 0) 0x7f1af01a5980 con 0x7f1af0106890 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 >> v1:172.21.15.138:6789/0 conn(0x7f1af010d300 legacy=0x7f1af01a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 >> v1:172.21.15.88:6789/0 conn(0x7f1af0109730 legacy=0x7f1af01a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af01a9040 con 0x7f1af0106890 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1af01a5bb0 con 0x7f1af0106890 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1af01a6080 con 0x7f1af0106890 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f1ae0001c90 con 0x7f1af0106890 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.433+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f1ae0005ba0 con 0x7f1af0106890 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.434+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad0005080 con 0x7f1af0106890 2024-03-26T21:55:36.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.434+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f1ae00038a0 con 0x7f1af0106890 2024-03-26T21:55:36.433 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.435+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(804..804 src has 268..804) ==== 7786+0+0 (unknown 2387307941 0 0) 0x7f1ae008a510 con 0x7f1af0106890 2024-03-26T21:55:36.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.440+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f1ae0057290 con 0x7f1af0106890 2024-03-26T21:55:36.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:36 smithi138 ceph-mon[33706]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T21:55:36.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:36 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]: dispatch 2024-03-26T21:55:36.485 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:36 smithi088 ceph-mon[30075]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T21:55:36.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:36 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]: dispatch 2024-03-26T21:55:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:36 smithi088 ceph-mon[39202]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T21:55:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:36 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]: dispatch 2024-03-26T21:55:36.614 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:36.619+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"} v 0) -- 0x7f1ad0005370 con 0x7f1af0106890 2024-03-26T21:55:37.223 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:37.227+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v805) ==== 223+0+0 (unknown 803965294 0 0) 0x7f1ae005af40 con 0x7f1af0106890 2024-03-26T21:55:37.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:37.348+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"} v 0) -- 0x7f1ad0001090 con 0x7f1af0106890 2024-03-26T21:55:37.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[30075]: pgmap v1738: 164 pgs: 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:55:37.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[30075]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:37.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]': finished 2024-03-26T21:55:37.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[30075]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T21:55:37.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]: dispatch 2024-03-26T21:55:37.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507250870' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:37.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[30075]: from='client.57266 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[39202]: pgmap v1738: 164 pgs: 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:55:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[39202]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]': finished 2024-03-26T21:55:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[39202]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T21:55:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]: dispatch 2024-03-26T21:55:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507250870' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:37 smithi088 ceph-mon[39202]: from='client.57266 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:37 smithi138 ceph-mon[33706]: pgmap v1738: 164 pgs: 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:55:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:37 smithi138 ceph-mon[33706]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:55:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]': finished 2024-03-26T21:55:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:37 smithi138 ceph-mon[33706]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T21:55:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/4118015100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pg_num": 12}]: dispatch 2024-03-26T21:55:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:37 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507250870' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:37 smithi138 ceph-mon[33706]: from='client.57266 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:38.245 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.249+0000 7f1aecff9640 1 -- 172.21.15.88:0/507250870 <== mon.2 v1:172.21.15.88:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v806) ==== 223+0+0 (unknown 4096091902 0 0) 0x7f1ae005f800 con 0x7f1af0106890 2024-03-26T21:55:38.245 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_objects = 10 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:55:38.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.252+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 >> v1:172.21.15.88:6800/3939589528 conn(0x7f1adc06f570 legacy=0x7f1adc071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:38.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.252+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 >> v1:172.21.15.88:6790/0 conn(0x7f1af0106890 legacy=0x7f1af010cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:38.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.252+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 shutdown_connections 2024-03-26T21:55:38.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.252+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 >> 172.21.15.88:0/507250870 conn(0x7f1af00fc350 msgr2=0x7f1af0100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:38.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.252+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 shutdown_connections 2024-03-26T21:55:38.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.252+0000 7f1af5439640 1 -- 172.21.15.88:0/507250870 wait complete. 2024-03-26T21:55:38.257 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool application enable 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 rados 2024-03-26T21:55:38.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.346+0000 7fb3872ed640 1 -- 172.21.15.88:0/1316702628 >> v1:172.21.15.88:6790/0 conn(0x7fb38010d2e0 legacy=0x7fb38010f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:38.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.346+0000 7fb3872ed640 1 -- 172.21.15.88:0/1316702628 shutdown_connections 2024-03-26T21:55:38.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.346+0000 7fb3872ed640 1 -- 172.21.15.88:0/1316702628 >> 172.21.15.88:0/1316702628 conn(0x7fb380075b80 msgr2=0x7fb380075f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:38.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.346+0000 7fb3872ed640 1 -- 172.21.15.88:0/1316702628 shutdown_connections 2024-03-26T21:55:38.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.346+0000 7fb3872ed640 1 -- 172.21.15.88:0/1316702628 wait complete. 2024-03-26T21:55:38.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.346+0000 7fb3872ed640 1 Processor -- start 2024-03-26T21:55:38.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.346+0000 7fb3872ed640 1 -- start start 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3872ed640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb3801a8c80 con 0x7fb38010d2e0 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3872ed640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb3801a9e80 con 0x7fb3801014d0 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3872ed640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb3801ab0c0 con 0x7fb380109770 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb384888640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fb380109770 0x7fb3801a3b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53662/0 (socket says 172.21.15.88:53662) 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb384888640 1 -- 172.21.15.88:0/425569014 learned_addr learned my addr 172.21.15.88:0/425569014 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb38588a640 1 --1- 172.21.15.88:0/425569014 >> v1:172.21.15.88:6789/0 conn(0x7fb38010d2e0 0x7fb3801a7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52662/0 (socket says 172.21.15.88:52662) 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 154837005 0 0) 0x7fb3801a8c80 con 0x7fb38010d2e0 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb36c003520 con 0x7fb38010d2e0 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519948874 0 0) 0x7fb3801ab0c0 con 0x7fb380109770 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3801a8c80 con 0x7fb380109770 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4167880069 0 0) 0x7fb36c003520 con 0x7fb38010d2e0 2024-03-26T21:55:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3801ab0c0 con 0x7fb38010d2e0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb37c004790 con 0x7fb38010d2e0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330202769 0 0) 0x7fb3801a9e80 con 0x7fb3801014d0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb36c003520 con 0x7fb3801014d0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4024340894 0 0) 0x7fb3801a8c80 con 0x7fb380109770 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3801a9e80 con 0x7fb380109770 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb3700018e0 con 0x7fb380109770 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3848697188 0 0) 0x7fb3801ab0c0 con 0x7fb38010d2e0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 >> v1:172.21.15.88:6790/0 conn(0x7fb380109770 legacy=0x7fb3801a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 >> v1:172.21.15.138:6789/0 conn(0x7fb3801014d0 legacy=0x7fb380105c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3801ac300 con 0x7fb38010d2e0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb3801ab2f0 con 0x7fb38010d2e0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb3801ab7c0 con 0x7fb38010d2e0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fb37c001eb0 con 0x7fb38010d2e0 2024-03-26T21:55:38.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.347+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fb37c004fe0 con 0x7fb38010d2e0 2024-03-26T21:55:38.347 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.348+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb364005080 con 0x7fb38010d2e0 2024-03-26T21:55:38.347 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.349+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fb37c003b50 con 0x7fb38010d2e0 2024-03-26T21:55:38.347 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.349+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(806..806 src has 268..806) ==== 7786+0+0 (unknown 2397008780 0 0) 0x7fb37c08a7d0 con 0x7fb38010d2e0 2024-03-26T21:55:38.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.354+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fb37c057590 con 0x7fb38010d2e0 2024-03-26T21:55:38.530 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:38.535+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"} v 0) -- 0x7fb364005370 con 0x7fb38010d2e0 2024-03-26T21:55:38.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:38 smithi138 ceph-mon[33706]: from='client.57266 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:55:38.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:38 smithi138 ceph-mon[33706]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T21:55:38.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:38 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/507250870' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:38.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:38 smithi138 ceph-mon[33706]: from='client.57266 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[39202]: from='client.57266 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[39202]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/507250870' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[39202]: from='client.57266 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[30075]: from='client.57266 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[30075]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/507250870' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:38.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:38 smithi088 ceph-mon[30075]: from='client.57266 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:55:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:39.244+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]=0 enabled application 'rados' on pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' v807) ==== 213+0+0 (unknown 3029240362 0 0) 0x7fb37c05b240 con 0x7fb38010d2e0 2024-03-26T21:55:39.354 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:39.358+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"} v 0) -- 0x7fb364001090 con 0x7fb38010d2e0 2024-03-26T21:55:39.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:39 smithi138 ceph-mon[33706]: pgmap v1741: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:55:39.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:39 smithi138 ceph-mon[33706]: from='client.57266 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:55:39.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:39 smithi138 ceph-mon[33706]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T21:55:39.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:39 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]: dispatch 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[39202]: pgmap v1741: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[39202]: from='client.57266 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[39202]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]: dispatch 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[30075]: pgmap v1741: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[30075]: from='client.57266 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[30075]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T21:55:39.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:39 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]: dispatch 2024-03-26T21:55:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:40.265+0000 7fb3767fc640 1 -- 172.21.15.88:0/425569014 <== mon.0 v1:172.21.15.88:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]=0 enabled application 'rados' on pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' v808) ==== 213+0+0 (unknown 3330837337 0 0) 0x7fb37c05fb00 con 0x7fb38010d2e0 2024-03-26T21:55:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:enabled application 'rados' on pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' 2024-03-26T21:55:40.263 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:40.267+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 >> v1:172.21.15.88:6800/3939589528 conn(0x7fb36c06f730 legacy=0x7fb36c071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:40.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:40.267+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 >> v1:172.21.15.88:6789/0 conn(0x7fb38010d2e0 legacy=0x7fb3801a7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:55:40.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:40.268+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 shutdown_connections 2024-03-26T21:55:40.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:40.268+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 >> 172.21.15.88:0/425569014 conn(0x7fb380075b80 msgr2=0x7fb380110af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:55:40.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:40.268+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 shutdown_connections 2024-03-26T21:55:40.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:55:40.268+0000 7fb3872ed640 1 -- 172.21.15.88:0/425569014 wait complete. 2024-03-26T21:55:40.277 INFO:tasks.workunit.client.0.smithi088.stderr:++ seq 1 10 2024-03-26T21:55:40.290 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.290 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj1 /etc/passwd 2024-03-26T21:55:40.328 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.328 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj2 /etc/passwd 2024-03-26T21:55:40.362 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.362 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj3 /etc/passwd 2024-03-26T21:55:40.395 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.395 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj4 /etc/passwd 2024-03-26T21:55:40.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj5 /etc/passwd 2024-03-26T21:55:40.462 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.462 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj6 /etc/passwd 2024-03-26T21:55:40.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.494 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj7 /etc/passwd 2024-03-26T21:55:40.527 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.527 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj8 /etc/passwd 2024-03-26T21:55:40.559 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.559 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj9 /etc/passwd 2024-03-26T21:55:40.592 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:55:40.592 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put obj10 /etc/passwd 2024-03-26T21:55:40.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ sleep 30 2024-03-26T21:55:40.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]': finished 2024-03-26T21:55:40.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:40 smithi138 ceph-mon[33706]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T21:55:40.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:40 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]: dispatch 2024-03-26T21:55:40.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]': finished 2024-03-26T21:55:40.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:40 smithi088 ceph-mon[30075]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T21:55:40.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:40 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]: dispatch 2024-03-26T21:55:40.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]': finished 2024-03-26T21:55:40.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:40 smithi088 ceph-mon[39202]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T21:55:40.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:40 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]: dispatch 2024-03-26T21:55:41.637 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:41 smithi088 ceph-mon[30075]: pgmap v1744: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:41.637 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:41 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]': finished 2024-03-26T21:55:41.637 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:41 smithi088 ceph-mon[30075]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T21:55:41.638 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:41 smithi088 ceph-mon[39202]: pgmap v1744: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:41.638 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:41 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]': finished 2024-03-26T21:55:41.638 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:41 smithi088 ceph-mon[39202]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T21:55:41.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:41 smithi138 ceph-mon[33706]: pgmap v1744: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 890 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:55:41.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:41 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/425569014' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "app": "rados"}]': finished 2024-03-26T21:55:41.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:41 smithi138 ceph-mon[33706]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T21:55:42.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:42.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:55:42 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:55:42.656 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:42.656 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:42.656 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:55:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:55:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:55:43.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:43 smithi138 ceph-mon[33706]: pgmap v1746: 176 pgs: 176 active+clean; 617 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 7.3 KiB/s wr, 1 op/s 2024-03-26T21:55:43.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:43.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:43 smithi088 ceph-mon[39202]: pgmap v1746: 176 pgs: 176 active+clean; 617 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 7.3 KiB/s wr, 1 op/s 2024-03-26T21:55:43.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:43.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:43 smithi088 ceph-mon[30075]: pgmap v1746: 176 pgs: 176 active+clean; 617 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 7.3 KiB/s wr, 1 op/s 2024-03-26T21:55:43.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:45.668 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:45 smithi138 ceph-mon[33706]: pgmap v1747: 176 pgs: 176 active+clean; 617 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:55:45.668 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:45 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:55:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:45 smithi088 ceph-mon[39202]: pgmap v1747: 176 pgs: 176 active+clean; 617 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:55:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:45 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:55:45.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:45 smithi088 ceph-mon[30075]: pgmap v1747: 176 pgs: 176 active+clean; 617 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:55:45.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:45 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:55:47.620 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[39202]: pgmap v1748: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T21:55:47.620 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[39202]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' is full (reached quota's max_objects: 10) 2024-03-26T21:55:47.620 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[39202]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:55:47.620 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[39202]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T21:55:47.621 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[30075]: pgmap v1748: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T21:55:47.621 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[30075]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' is full (reached quota's max_objects: 10) 2024-03-26T21:55:47.621 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[30075]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:55:47.621 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:47 smithi088 ceph-mon[30075]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T21:55:47.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:47 smithi138 ceph-mon[33706]: pgmap v1748: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T21:55:47.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:47 smithi138 ceph-mon[33706]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' is full (reached quota's max_objects: 10) 2024-03-26T21:55:47.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:47 smithi138 ceph-mon[33706]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:55:47.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:47 smithi138 ceph-mon[33706]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T21:55:49.227 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:48 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:49.227 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:48 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:49.227 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:48 smithi138 ceph-mon[33706]: pgmap v1750: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T21:55:49.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:48 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:49.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:48 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:49.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:48 smithi088 ceph-mon[39202]: pgmap v1750: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T21:55:49.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:48 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:49.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:48 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:49.236 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:48 smithi088 ceph-mon[30075]: pgmap v1750: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T21:55:50.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:55:50.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:55:50.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:50 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:50.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:55:50.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:55:50.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:50 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:51.726 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:51 smithi138 ceph-mon[33706]: pgmap v1751: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T21:55:51.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:51 smithi088 ceph-mon[39202]: pgmap v1751: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T21:55:51.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:51 smithi088 ceph-mon[30075]: pgmap v1751: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T21:55:52.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:55:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:55:52.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:55:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:55:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:55:53.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:53 smithi138 ceph-mon[33706]: pgmap v1752: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-26T21:55:53.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:53.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:53 smithi088 ceph-mon[30075]: pgmap v1752: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-26T21:55:53.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:53.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:53 smithi088 ceph-mon[39202]: pgmap v1752: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-26T21:55:53.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:55:55.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:55 smithi138 ceph-mon[33706]: pgmap v1753: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-26T21:55:55.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:55 smithi088 ceph-mon[30075]: pgmap v1753: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-26T21:55:55.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:55 smithi088 ceph-mon[39202]: pgmap v1753: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-26T21:55:57.476 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:57 smithi138 ceph-mon[33706]: pgmap v1754: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:57.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:57 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:57.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:57.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:57 smithi088 ceph-mon[30075]: pgmap v1754: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:57.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:57 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:57.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:57.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:57 smithi088 ceph-mon[39202]: pgmap v1754: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:55:57.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:57 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:55:57.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:55:59.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:55:59 smithi138 ceph-mon[33706]: pgmap v1755: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-26T21:55:59.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:55:59 smithi088 ceph-mon[30075]: pgmap v1755: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-26T21:55:59.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:55:59 smithi088 ceph-mon[39202]: pgmap v1755: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-26T21:56:01.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:01 smithi138 ceph-mon[33706]: pgmap v1756: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:01.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:01 smithi088 ceph-mon[30075]: pgmap v1756: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:01.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:01 smithi088 ceph-mon[39202]: pgmap v1756: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:02.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:56:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:56:02.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:56:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:56:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:56:03.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:03 smithi138 ceph-mon[33706]: pgmap v1757: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:03.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:03.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:03 smithi088 ceph-mon[30075]: pgmap v1757: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:03.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:03.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:03 smithi088 ceph-mon[39202]: pgmap v1757: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:03.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:05.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:05 smithi138 ceph-mon[33706]: pgmap v1758: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:05 smithi088 ceph-mon[39202]: pgmap v1758: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:05.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:05 smithi088 ceph-mon[30075]: pgmap v1758: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:07.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:07 smithi138 ceph-mon[33706]: pgmap v1759: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:07.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:07 smithi088 ceph-mon[39202]: pgmap v1759: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:07.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:07 smithi088 ceph-mon[30075]: pgmap v1759: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:09.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:09 smithi138 ceph-mon[33706]: pgmap v1760: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:09.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:09 smithi088 ceph-mon[30075]: pgmap v1760: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:09.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:09 smithi088 ceph-mon[39202]: pgmap v1760: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:10.626 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=156074 2024-03-26T21:56:10.626 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 max_objects 100 2024-03-26T21:56:10.627 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put onemore /etc/passwd 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- 172.21.15.88:0/1020169806 >> v1:172.21.15.88:6789/0 conn(0x7fec2410a970 legacy=0x7fec2410cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- 172.21.15.88:0/1020169806 shutdown_connections 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- 172.21.15.88:0/1020169806 >> 172.21.15.88:0/1020169806 conn(0x7fec240fc370 msgr2=0x7fec240fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- 172.21.15.88:0/1020169806 shutdown_connections 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- 172.21.15.88:0/1020169806 wait complete. 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 Processor -- start 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- start start 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec241a8c50 con 0x7fec2410e480 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec241a9e90 con 0x7fec24102b70 2024-03-26T21:56:10.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.725+0000 7fec2af56640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec241ab0d0 con 0x7fec2410a970 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec23fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fec2410a970 0x7fec241a3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:33430/0 (socket says 172.21.15.88:33430) 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec23fff640 1 -- 172.21.15.88:0/907618091 learned_addr learned my addr 172.21.15.88:0/907618091 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1976760789 0 0) 0x7fec241a8c50 con 0x7fec2410e480 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec14003160 con 0x7fec2410e480 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485494636 0 0) 0x7fec241ab0d0 con 0x7fec2410a970 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec241a8c50 con 0x7fec2410a970 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3304322077 0 0) 0x7fec241a9e90 con 0x7fec24102b70 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec241ab0d0 con 0x7fec24102b70 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 341346067 0 0) 0x7fec14003160 con 0x7fec2410e480 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec241a9e90 con 0x7fec2410e480 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fec1c001c30 con 0x7fec2410e480 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3416696988 0 0) 0x7fec241a8c50 con 0x7fec2410a970 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec14003160 con 0x7fec2410a970 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fec10004420 con 0x7fec2410a970 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 171558808 0 0) 0x7fec241ab0d0 con 0x7fec24102b70 2024-03-26T21:56:10.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec241a8c50 con 0x7fec24102b70 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fec180013d0 con 0x7fec24102b70 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3474110730 0 0) 0x7fec241a9e90 con 0x7fec2410e480 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 >> v1:172.21.15.88:6790/0 conn(0x7fec2410a970 legacy=0x7fec241a3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 >> v1:172.21.15.138:6789/0 conn(0x7fec24102b70 legacy=0x7fec2410a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec241ac310 con 0x7fec2410e480 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec241ab300 con 0x7fec2410e480 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fec1c003e20 con 0x7fec2410e480 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.726+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fec1c004e00 con 0x7fec2410e480 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.727+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fec241ab7d0 con 0x7fec2410e480 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.728+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec2406a8f0 con 0x7fec2410e480 2024-03-26T21:56:10.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.728+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fec1c003990 con 0x7fec2410e480 2024-03-26T21:56:10.727 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.728+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(809..809 src has 268..809) ==== 7799+0+0 (unknown 2268926396 0 0) 0x7fec1c089480 con 0x7fec2410e480 2024-03-26T21:56:10.727 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.729+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=810}) -- 0x7fec241a9e90 con 0x7fec2410e480 2024-03-26T21:56:10.729 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.733+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fec1c056330 con 0x7fec2410e480 2024-03-26T21:56:10.907 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:10.911+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"} v 0) -- 0x7fec24068c20 con 0x7fec2410e480 2024-03-26T21:56:11.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:11.222+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v810) ==== 225+0+0 (unknown 1563680987 0 0) 0x7fec1c059fe0 con 0x7fec2410e480 2024-03-26T21:56:11.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:11.222+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 11 ==== osd_map(810..810 src has 268..810) ==== 628+0+0 (unknown 2727196258 0 0) 0x7fec1c004540 con 0x7fec2410e480 2024-03-26T21:56:11.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:11.223+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=811}) -- 0x7fec14003160 con 0x7fec2410e480 2024-03-26T21:56:11.330 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:11.334+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"} v 0) -- 0x7fec2410bed0 con 0x7fec2410e480 2024-03-26T21:56:11.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:11 smithi088 ceph-mon[39202]: pgmap v1761: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:11.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:11 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T21:56:11.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:11 smithi088 ceph-mon[30075]: pgmap v1761: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:11.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:11 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T21:56:11.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:11 smithi138 ceph-mon[33706]: pgmap v1761: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:11.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:11 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T21:56:12.104 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.107+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v811) ==== 225+0+0 (unknown 1976622243 0 0) 0x7fec1c05e8a0 con 0x7fec2410e480 2024-03-26T21:56:12.104 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_objects = 100 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:56:12.104 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.108+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 <== mon.0 v1:172.21.15.88:6789/0 13 ==== osd_map(811..811 src has 268..811) ==== 628+0+0 (unknown 3476090861 0 0) 0x7fec1c051c70 con 0x7fec2410e480 2024-03-26T21:56:12.104 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.108+0000 7fec21ffb640 1 -- 172.21.15.88:0/907618091 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=812}) -- 0x7fec241ab0d0 con 0x7fec2410e480 2024-03-26T21:56:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.112+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 >> v1:172.21.15.88:6800/3939589528 conn(0x7fec1406f4e0 legacy=0x7fec140719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:12.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.112+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 >> v1:172.21.15.88:6789/0 conn(0x7fec2410e480 legacy=0x7fec241a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:12.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.112+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 shutdown_connections 2024-03-26T21:56:12.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.112+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 >> 172.21.15.88:0/907618091 conn(0x7fec240fc370 msgr2=0x7fec240fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:12.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.112+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 shutdown_connections 2024-03-26T21:56:12.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.113+0000 7fec2af56640 1 -- 172.21.15.88:0/907618091 wait complete. 2024-03-26T21:56:12.124 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 156074 2024-03-26T21:56:12.124 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T21:56:12.124 INFO:tasks.workunit.client.0.smithi088.stderr:+ true 2024-03-26T21:56:12.124 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put twomore /etc/passwd 2024-03-26T21:56:12.159 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 max_bytes 100 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]': finished 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' no longer out of quota; removing NO_QUOTA flag 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]': finished 2024-03-26T21:56:12.240 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-mon[33706]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T21:56:12.243 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.247+0000 7f37c1356640 1 -- 172.21.15.88:0/488494068 >> v1:172.21.15.88:6790/0 conn(0x7f37bc10d320 legacy=0x7f37bc10f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:12.243 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.247+0000 7f37c1356640 1 -- 172.21.15.88:0/488494068 shutdown_connections 2024-03-26T21:56:12.243 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.247+0000 7f37c1356640 1 -- 172.21.15.88:0/488494068 >> 172.21.15.88:0/488494068 conn(0x7f37bc0fc3b0 msgr2=0x7f37bc0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:12.243 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.247+0000 7f37c1356640 1 -- 172.21.15.88:0/488494068 shutdown_connections 2024-03-26T21:56:12.244 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.247+0000 7f37c1356640 1 -- 172.21.15.88:0/488494068 wait complete. 2024-03-26T21:56:12.244 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.247+0000 7f37c1356640 1 Processor -- start 2024-03-26T21:56:12.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37c1356640 1 -- start start 2024-03-26T21:56:12.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37c1356640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37bc1a69c0 con 0x7f37bc109750 2024-03-26T21:56:12.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37c1356640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37bc1a7c00 con 0x7f37bc10d320 2024-03-26T21:56:12.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37c1356640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37bc1a8e40 con 0x7f37bc1068b0 2024-03-26T21:56:12.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37baffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f37bc1068b0 0x7f37bc10cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:33488/0 (socket says 172.21.15.88:33488) 2024-03-26T21:56:12.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37baffd640 1 -- 172.21.15.88:0/231650221 learned_addr learned my addr 172.21.15.88:0/231650221 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182927326 0 0) 0x7f37bc1a8e40 con 0x7f37bc1068b0 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37a8003520 con 0x7f37bc1068b0 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134501854 0 0) 0x7f37bc1a69c0 con 0x7f37bc109750 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37bc1a8e40 con 0x7f37bc109750 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1057030432 0 0) 0x7f37a8003520 con 0x7f37bc1068b0 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37bc1a69c0 con 0x7f37bc1068b0 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f37b4001650 con 0x7f37bc1068b0 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 598227041 0 0) 0x7f37bc1a8e40 con 0x7f37bc109750 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37a8003520 con 0x7f37bc109750 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f37a4001830 con 0x7f37bc109750 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474140447 0 0) 0x7f37bc1a7c00 con 0x7f37bc10d320 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37bc1a8e40 con 0x7f37bc10d320 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1939582904 0 0) 0x7f37bc1a69c0 con 0x7f37bc1068b0 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 >> v1:172.21.15.138:6789/0 conn(0x7f37bc10d320 legacy=0x7f37bc1a50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:12.247 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.248+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 >> v1:172.21.15.88:6789/0 conn(0x7f37bc109750 legacy=0x7f37bc1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.249+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc1aa080 con 0x7f37bc1068b0 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.249+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc1a9070 con 0x7f37bc1068b0 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.249+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f37bc1a9540 con 0x7f37bc1068b0 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.249+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f37b4003e00 con 0x7f37bc1068b0 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.249+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f37b4005e70 con 0x7f37bc1068b0 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.249+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f379c0050b0 con 0x7f37bc1068b0 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.250+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f37b4003970 con 0x7f37bc1068b0 2024-03-26T21:56:12.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.251+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(811..811 src has 268..811) ==== 7799+0+0 (unknown 1718475253 0 0) 0x7f37b408a6d0 con 0x7f37bc1068b0 2024-03-26T21:56:12.251 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.255+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f37b40574a0 con 0x7f37bc1068b0 2024-03-26T21:56:12.434 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:12.438+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"} v 0) -- 0x7f379c0053a0 con 0x7f37bc1068b0 2024-03-26T21:56:12.487 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:56:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:56:12.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]': finished 2024-03-26T21:56:12.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T21:56:12.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T21:56:12.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:12.578 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' no longer out of quota; removing NO_QUOTA flag 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]': finished 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[39202]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]': finished 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' no longer out of quota; removing NO_QUOTA flag 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/907618091' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "100"}]': finished 2024-03-26T21:56:12.579 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:12 smithi088 ceph-mon[30075]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T21:56:12.580 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:56:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:56:13.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:13.239+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v812) ==== 221+0+0 (unknown 4031951718 0 0) 0x7f37b405b150 con 0x7f37bc1068b0 2024-03-26T21:56:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:13.354+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"} v 0) -- 0x7f379c005ae0 con 0x7f37bc1068b0 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[39202]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/231650221' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[39202]: from='client.57392 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[30075]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/231650221' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:13.498 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:13 smithi088 ceph-mon[30075]: from='client.57392 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:13.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:13 smithi138 ceph-mon[33706]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:13.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:13.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:13 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/231650221' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:13.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:13 smithi138 ceph-mon[33706]: from='client.57392 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:14.251 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:14.254+0000 7f37b3fff640 1 -- 172.21.15.88:0/231650221 <== mon.2 v1:172.21.15.88:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v813) ==== 221+0+0 (unknown 1879204487 0 0) 0x7f37b405fa10 con 0x7f37bc1068b0 2024-03-26T21:56:14.251 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_bytes = 100 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:56:14.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:14.257+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 >> v1:172.21.15.88:6800/3939589528 conn(0x7f37a806f5d0 legacy=0x7f37a8071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:14.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:14.257+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 >> v1:172.21.15.88:6790/0 conn(0x7f37bc1068b0 legacy=0x7f37bc10cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:14.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:14.258+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 shutdown_connections 2024-03-26T21:56:14.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:14.258+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 >> 172.21.15.88:0/231650221 conn(0x7f37bc0fc3b0 msgr2=0x7f37bc10ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:14.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:14.258+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 shutdown_connections 2024-03-26T21:56:14.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:14.258+0000 7f37c1356640 1 -- 172.21.15.88:0/231650221 wait complete. 2024-03-26T21:56:14.265 INFO:tasks.workunit.client.0.smithi088.stderr:+ sleep 30 2024-03-26T21:56:14.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:14 smithi138 ceph-mon[33706]: from='client.57392 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T21:56:14.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:14 smithi138 ceph-mon[33706]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T21:56:14.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:14 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/231650221' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:14.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:14 smithi138 ceph-mon[33706]: from='client.57392 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:14.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[30075]: from='client.57392 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T21:56:14.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[30075]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T21:56:14.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/231650221' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:14.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[30075]: from='client.57392 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:14.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[39202]: from='client.57392 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T21:56:14.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[39202]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T21:56:14.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/231650221' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:14.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:14 smithi088 ceph-mon[39202]: from='client.57392 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T21:56:15.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:15 smithi138 ceph-mon[33706]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:15.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:15 smithi138 ceph-mon[33706]: from='client.57392 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T21:56:15.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:15 smithi138 ceph-mon[33706]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T21:56:15.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:15 smithi088 ceph-mon[39202]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:15.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:15 smithi088 ceph-mon[39202]: from='client.57392 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T21:56:15.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:15 smithi088 ceph-mon[39202]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T21:56:15.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:15 smithi088 ceph-mon[30075]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 878 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:15.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:15 smithi088 ceph-mon[30075]: from='client.57392 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T21:56:15.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:15 smithi088 ceph-mon[30075]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T21:56:17.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:17 smithi138 ceph-mon[33706]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-26T21:56:17.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:17 smithi138 ceph-mon[33706]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' is full (reached quota's max_bytes: 100 B) 2024-03-26T21:56:17.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:17 smithi138 ceph-mon[33706]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:56:17.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:17 smithi138 ceph-mon[33706]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T21:56:17.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[30075]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-26T21:56:17.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[30075]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' is full (reached quota's max_bytes: 100 B) 2024-03-26T21:56:17.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[30075]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:56:17.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[30075]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T21:56:17.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[39202]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-03-26T21:56:17.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[39202]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' is full (reached quota's max_bytes: 100 B) 2024-03-26T21:56:17.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[39202]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:56:17.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:17 smithi088 ceph-mon[39202]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T21:56:19.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:19 smithi138 ceph-mon[33706]: pgmap v1770: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T21:56:19.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:19 smithi088 ceph-mon[30075]: pgmap v1770: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T21:56:19.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:19 smithi088 ceph-mon[39202]: pgmap v1770: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T21:56:21.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:21 smithi138 ceph-mon[33706]: pgmap v1771: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 740 B/s rd, 3.5 KiB/s wr, 1 op/s 2024-03-26T21:56:21.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:21 smithi088 ceph-mon[39202]: pgmap v1771: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 740 B/s rd, 3.5 KiB/s wr, 1 op/s 2024-03-26T21:56:21.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:21 smithi088 ceph-mon[30075]: pgmap v1771: 176 pgs: 176 active+clean; 721 KiB data, 879 MiB used, 714 GiB / 715 GiB avail; 740 B/s rd, 3.5 KiB/s wr, 1 op/s 2024-03-26T21:56:22.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:56:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:56:22.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:56:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:56:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:56:23.566 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:23 smithi088 ceph-mon[39202]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T21:56:23.566 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:23.566 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:23 smithi088 ceph-mon[30075]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T21:56:23.566 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:23.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:23 smithi138 ceph-mon[33706]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T21:56:23.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:25.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:25 smithi138 ceph-mon[33706]: pgmap v1773: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T21:56:25.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:25 smithi088 ceph-mon[39202]: pgmap v1773: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T21:56:25.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:25 smithi088 ceph-mon[30075]: pgmap v1773: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T21:56:27.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:27 smithi138 ceph-mon[33706]: pgmap v1774: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:56:27.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:27.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:27 smithi088 ceph-mon[39202]: pgmap v1774: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:56:27.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:27.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:27 smithi088 ceph-mon[30075]: pgmap v1774: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:56:27.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:29.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:29 smithi138 ceph-mon[33706]: pgmap v1775: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-26T21:56:29.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:29 smithi088 ceph-mon[39202]: pgmap v1775: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-26T21:56:29.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:29 smithi088 ceph-mon[30075]: pgmap v1775: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-26T21:56:31.528 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:31 smithi138 ceph-mon[33706]: pgmap v1776: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:31.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:31 smithi088 ceph-mon[39202]: pgmap v1776: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:31.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:31 smithi088 ceph-mon[30075]: pgmap v1776: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:32.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:56:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:56:32.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:56:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:56:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:56:33.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:33 smithi138 ceph-mon[33706]: pgmap v1777: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:33.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:33.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:33 smithi088 ceph-mon[39202]: pgmap v1777: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:33.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:33.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:33 smithi088 ceph-mon[30075]: pgmap v1777: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:33.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:35.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:35 smithi138 ceph-mon[33706]: pgmap v1778: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:35.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:35 smithi088 ceph-mon[39202]: pgmap v1778: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:35.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:35 smithi088 ceph-mon[30075]: pgmap v1778: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:37 smithi138 ceph-mon[33706]: pgmap v1779: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:37.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:37 smithi088 ceph-mon[30075]: pgmap v1779: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:37.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:37 smithi088 ceph-mon[39202]: pgmap v1779: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:39.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:39 smithi138 ceph-mon[33706]: pgmap v1780: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:39.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:39 smithi088 ceph-mon[39202]: pgmap v1780: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:39.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:39 smithi088 ceph-mon[30075]: pgmap v1780: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:41.539 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:41 smithi088 ceph-mon[30075]: pgmap v1781: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:41.539 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:41 smithi088 ceph-mon[39202]: pgmap v1781: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:41.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:41 smithi138 ceph-mon[33706]: pgmap v1781: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:42.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:56:42 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:56:42.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:42.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:42.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:42.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:56:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:56:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:56:43.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:43 smithi138 ceph-mon[33706]: pgmap v1782: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:43.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:43 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:43.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:43 smithi088 ceph-mon[30075]: pgmap v1782: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:43.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:43 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:43.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:43 smithi088 ceph-mon[39202]: pgmap v1782: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:43.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:43 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:44.267 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=156175 2024-03-26T21:56:44.267 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 max_bytes 0 2024-03-26T21:56:44.267 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put two /etc/passwd 2024-03-26T21:56:44.361 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.365+0000 7f80f4e7b640 1 -- 172.21.15.88:0/167470712 >> v1:172.21.15.88:6789/0 conn(0x7f80f0109750 legacy=0x7f80f010bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:44.362 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.365+0000 7f80f4e7b640 1 -- 172.21.15.88:0/167470712 shutdown_connections 2024-03-26T21:56:44.362 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.365+0000 7f80f4e7b640 1 -- 172.21.15.88:0/167470712 >> 172.21.15.88:0/167470712 conn(0x7f80f00fc370 msgr2=0x7f80f00fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:44.362 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.365+0000 7f80f4e7b640 1 -- 172.21.15.88:0/167470712 shutdown_connections 2024-03-26T21:56:44.362 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.365+0000 7f80f4e7b640 1 -- 172.21.15.88:0/167470712 wait complete. 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80f4e7b640 1 Processor -- start 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80f4e7b640 1 -- start start 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80f4e7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80f01a6920 con 0x7f80f0109750 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80f4e7b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80f01a7b20 con 0x7f80f01068b0 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80f4e7b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f80f01a8d60 con 0x7f80f010d320 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80edd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f80f0109750 0x7f80f01a1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60560/0 (socket says 172.21.15.88:60560) 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80edd9b640 1 -- 172.21.15.88:0/3742931714 learned_addr learned my addr 172.21.15.88:0/3742931714 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018336007 0 0) 0x7f80f01a6920 con 0x7f80f0109750 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80d8004920 con 0x7f80f0109750 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213360548 0 0) 0x7f80f01a8d60 con 0x7f80f010d320 2024-03-26T21:56:44.365 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.366+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80f01a6920 con 0x7f80f010d320 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940920594 0 0) 0x7f80f01a7b20 con 0x7f80f01068b0 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80f01a8d60 con 0x7f80f01068b0 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4102545774 0 0) 0x7f80d8004920 con 0x7f80f0109750 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80f01a7b20 con 0x7f80f0109750 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f80e4004470 con 0x7f80f0109750 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3981676048 0 0) 0x7f80f01a6920 con 0x7f80f010d320 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80d8004920 con 0x7f80f010d320 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f80e8001470 con 0x7f80f010d320 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 856650878 0 0) 0x7f80f01a8d60 con 0x7f80f01068b0 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80f01a6920 con 0x7f80f01068b0 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f80e0001870 con 0x7f80f01068b0 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 832072935 0 0) 0x7f80f01a7b20 con 0x7f80f0109750 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 >> v1:172.21.15.88:6790/0 conn(0x7f80f010d320 legacy=0x7f80f01a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 >> v1:172.21.15.138:6789/0 conn(0x7f80f01068b0 legacy=0x7f80f010cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f01a9fa0 con 0x7f80f0109750 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f80f01a8f30 con 0x7f80f0109750 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f80f01a9400 con 0x7f80f0109750 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f80e4001b40 con 0x7f80f0109750 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.367+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f80e4004dd0 con 0x7f80f0109750 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.368+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d00050b0 con 0x7f80f0109750 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.369+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f80e4003680 con 0x7f80f0109750 2024-03-26T21:56:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.369+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(814..814 src has 268..814) ==== 7799+0+0 (unknown 1400791242 0 0) 0x7f80e408a520 con 0x7f80f0109750 2024-03-26T21:56:44.368 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.369+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=815}) -- 0x7f80f01a7b20 con 0x7f80f0109750 2024-03-26T21:56:44.370 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.374+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f80e4057290 con 0x7f80f0109750 2024-03-26T21:56:44.547 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:44.551+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"} v 0) -- 0x7f80d00053a0 con 0x7f80f0109750 2024-03-26T21:56:45.296 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:45.300+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v815) ==== 217+0+0 (unknown 3487437681 0 0) 0x7f80e405af40 con 0x7f80f0109750 2024-03-26T21:56:45.297 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:45.300+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 11 ==== osd_map(815..815 src has 268..815) ==== 628+0+0 (unknown 2804061620 0 0) 0x7f80e40882d0 con 0x7f80f0109750 2024-03-26T21:56:45.297 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:45.300+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=816}) -- 0x7f80f01a8d60 con 0x7f80f0109750 2024-03-26T21:56:45.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:45.414+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"} v 0) -- 0x7f80d0005ae0 con 0x7f80f0109750 2024-03-26T21:56:45.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:45 smithi138 ceph-mon[33706]: pgmap v1783: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:45.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:45 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:56:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:45 smithi088 ceph-mon[39202]: pgmap v1783: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:45 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:56:45.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:45 smithi088 ceph-mon[30075]: pgmap v1783: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:56:45.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:45 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:56:46.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.315+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v816) ==== 217+0+0 (unknown 1287143922 0 0) 0x7f80e405f800 con 0x7f80f0109750 2024-03-26T21:56:46.312 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_bytes = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:56:46.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.316+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 <== mon.0 v1:172.21.15.88:6789/0 13 ==== osd_map(816..816 src has 268..816) ==== 628+0+0 (unknown 2923071285 0 0) 0x7f80e4052bd0 con 0x7f80f0109750 2024-03-26T21:56:46.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.316+0000 7f80df7fe640 1 -- 172.21.15.88:0/3742931714 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=817}) -- 0x7f80d807c7e0 con 0x7f80f0109750 2024-03-26T21:56:46.315 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.319+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 >> v1:172.21.15.88:6800/3939589528 conn(0x7f80d806f6b0 legacy=0x7f80d8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.319+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 >> v1:172.21.15.88:6789/0 conn(0x7f80f0109750 legacy=0x7f80f01a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.319+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 shutdown_connections 2024-03-26T21:56:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.319+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 >> 172.21.15.88:0/3742931714 conn(0x7f80f00fc370 msgr2=0x7f80f00fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.319+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 shutdown_connections 2024-03-26T21:56:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.319+0000 7f80f4e7b640 1 -- 172.21.15.88:0/3742931714 wait complete. 2024-03-26T21:56:46.329 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 max_objects 0 2024-03-26T21:56:46.416 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.419+0000 7fbc391e9640 1 -- 172.21.15.88:0/438913400 >> v1:172.21.15.88:6790/0 conn(0x7fbc3410a950 legacy=0x7fbc3410cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.416 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.419+0000 7fbc391e9640 1 -- 172.21.15.88:0/438913400 shutdown_connections 2024-03-26T21:56:46.416 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.419+0000 7fbc391e9640 1 -- 172.21.15.88:0/438913400 >> 172.21.15.88:0/438913400 conn(0x7fbc340fc350 msgr2=0x7fbc340fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:46.416 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.420+0000 7fbc391e9640 1 -- 172.21.15.88:0/438913400 shutdown_connections 2024-03-26T21:56:46.416 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.420+0000 7fbc391e9640 1 -- 172.21.15.88:0/438913400 wait complete. 2024-03-26T21:56:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.420+0000 7fbc391e9640 1 Processor -- start 2024-03-26T21:56:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.420+0000 7fbc391e9640 1 -- start start 2024-03-26T21:56:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc391e9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc341a6a00 con 0x7fbc3410e460 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc391e9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc341a7c40 con 0x7fbc34102b50 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc391e9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc341a8e80 con 0x7fbc3410a950 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc3259c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbc3410a950 0x7fbc341a1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42546/0 (socket says 172.21.15.88:42546) 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc3259c640 1 -- 172.21.15.88:0/3245936251 learned_addr learned my addr 172.21.15.88:0/3245936251 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618956997 0 0) 0x7fbc341a6a00 con 0x7fbc3410e460 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc1c003520 con 0x7fbc3410e460 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407271444 0 0) 0x7fbc341a8e80 con 0x7fbc3410a950 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc341a6a00 con 0x7fbc3410a950 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2686216309 0 0) 0x7fbc341a7c40 con 0x7fbc34102b50 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc341a8e80 con 0x7fbc34102b50 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3557319149 0 0) 0x7fbc1c003520 con 0x7fbc3410e460 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc341a7c40 con 0x7fbc3410e460 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3005411962 0 0) 0x7fbc341a6a00 con 0x7fbc3410a950 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc1c003520 con 0x7fbc3410a950 2024-03-26T21:56:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fbc2c001470 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fbc240043f0 con 0x7fbc3410a950 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4100233481 0 0) 0x7fbc341a8e80 con 0x7fbc34102b50 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc341a6a00 con 0x7fbc34102b50 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fbc280017f0 con 0x7fbc34102b50 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1341194272 0 0) 0x7fbc341a7c40 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 >> v1:172.21.15.88:6790/0 conn(0x7fbc3410a950 legacy=0x7fbc341a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 >> v1:172.21.15.138:6789/0 conn(0x7fbc34102b50 legacy=0x7fbc3410a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc341aa0c0 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.421+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc341a90b0 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.422+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc341a9580 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.422+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fbc2c003e00 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.422+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fbc2c004c70 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.423+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc20005080 con 0x7fbc3410e460 2024-03-26T21:56:46.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.423+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fbc2c01b850 con 0x7fbc3410e460 2024-03-26T21:56:46.422 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.423+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(816..816 src has 268..816) ==== 7799+0+0 (unknown 3977770568 0 0) 0x7fbc2c089450 con 0x7fbc3410e460 2024-03-26T21:56:46.422 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.423+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=817}) -- 0x7fbc341a7c40 con 0x7fbc3410e460 2024-03-26T21:56:46.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.428+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fbc2c0561c0 con 0x7fbc3410e460 2024-03-26T21:56:46.603 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:46.606+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"} v 0) -- 0x7fbc20005370 con 0x7fbc3410e460 2024-03-26T21:56:46.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:56:46.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:46 smithi138 ceph-mon[33706]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T21:56:46.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:46 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:56:46.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:56:46.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:46 smithi088 ceph-mon[39202]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T21:56:46.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:46 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:56:46.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:56:46.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:46 smithi088 ceph-mon[30075]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T21:56:46.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:46 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:56:47.112 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:47.115+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v817) ==== 221+0+0 (unknown 586855771 0 0) 0x7fbc2c059e70 con 0x7fbc3410e460 2024-03-26T21:56:47.112 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:47.115+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 11 ==== osd_map(817..817 src has 268..817) ==== 628+0+0 (unknown 3082660764 0 0) 0x7fbc2c087200 con 0x7fbc3410e460 2024-03-26T21:56:47.112 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:47.115+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=818}) -- 0x7fbc1c003520 con 0x7fbc3410e460 2024-03-26T21:56:47.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:47.229+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"} v 0) -- 0x7fbc20000c40 con 0x7fbc3410e460 2024-03-26T21:56:47.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: pgmap v1785: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:56:47.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:56:47.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T21:56:47.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:56:47.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' no longer out of quota; removing NO_QUOTA flag 2024-03-26T21:56:47.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:56:47.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:56:47.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T21:56:47.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:47 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:56:47.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: pgmap v1785: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:56:47.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' no longer out of quota; removing NO_QUOTA flag 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: pgmap v1785: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3742931714' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' no longer out of quota; removing NO_QUOTA flag 2024-03-26T21:56:47.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:56:47.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:56:47.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T21:56:47.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:47 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:56:48.113 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.116+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v818) ==== 221+0+0 (unknown 2816979337 0 0) 0x7fbc2c05e730 con 0x7fbc3410e460 2024-03-26T21:56:48.113 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_objects = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:56:48.114 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.117+0000 7fbc1bfff640 1 -- 172.21.15.88:0/3245936251 <== mon.0 v1:172.21.15.88:6789/0 13 ==== osd_map(818..818 src has 268..818) ==== 628+0+0 (unknown 3900631367 0 0) 0x7fbc2c051b00 con 0x7fbc3410e460 2024-03-26T21:56:48.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.120+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 >> v1:172.21.15.88:6800/3939589528 conn(0x7fbc1c06f5d0 legacy=0x7fbc1c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:48.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.121+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 >> v1:172.21.15.88:6789/0 conn(0x7fbc3410e460 legacy=0x7fbc341a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:48.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.121+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 shutdown_connections 2024-03-26T21:56:48.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.121+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 >> 172.21.15.88:0/3245936251 conn(0x7fbc340fc350 msgr2=0x7fbc34110870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:48.118 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.121+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 shutdown_connections 2024-03-26T21:56:48.118 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.121+0000 7fbc391e9640 1 -- 172.21.15.88:0/3245936251 wait complete. 2024-03-26T21:56:48.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 156175 2024-03-26T21:56:48.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T21:56:48.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ true 2024-03-26T21:56:48.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put three /etc/passwd 2024-03-26T21:56:48.166 INFO:tasks.workunit.client.0.smithi088.stderr:++ uuidgen 2024-03-26T21:56:48.167 INFO:tasks.workunit.client.0.smithi088.stderr:+ pp=4328d745-f987-428c-ba08-f658a5eea7e4 2024-03-26T21:56:48.168 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool create 4328d745-f987-428c-ba08-f658a5eea7e4 12 2024-03-26T21:56:48.253 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.254+0000 7efc1b916640 1 -- 172.21.15.88:0/2287876149 >> v1:172.21.15.88:6790/0 conn(0x7efc14107990 legacy=0x7efc14107d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:48.253 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.254+0000 7efc1b916640 1 -- 172.21.15.88:0/2287876149 shutdown_connections 2024-03-26T21:56:48.253 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.254+0000 7efc1b916640 1 -- 172.21.15.88:0/2287876149 >> 172.21.15.88:0/2287876149 conn(0x7efc140fd430 msgr2=0x7efc140ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:48.253 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.254+0000 7efc1b916640 1 -- 172.21.15.88:0/2287876149 shutdown_connections 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.254+0000 7efc1b916640 1 -- 172.21.15.88:0/2287876149 wait complete. 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc1b916640 1 Processor -- start 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc1b916640 1 -- start start 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc1b916640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc141a6ae0 con 0x7efc1410a830 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc1b916640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc141a7ce0 con 0x7efc14107990 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc1b916640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc141a8f20 con 0x7efc1410e400 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc18eb1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efc1410a830 0x7efc141a1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60632/0 (socket says 172.21.15.88:60632) 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc19eb3640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7efc1410e400 0x7efc141a51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42594/0 (socket says 172.21.15.88:42594) 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc18eb1640 1 -- 172.21.15.88:0/2568962121 learned_addr learned my addr 172.21.15.88:0/2568962121 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2748854418 0 0) 0x7efc141a6ae0 con 0x7efc1410a830 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc00003520 con 0x7efc1410a830 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288580941 0 0) 0x7efc141a8f20 con 0x7efc1410e400 2024-03-26T21:56:48.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc141a6ae0 con 0x7efc1410e400 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 551502074 0 0) 0x7efc00003520 con 0x7efc1410a830 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc141a8f20 con 0x7efc1410a830 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.255+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7efc04001930 con 0x7efc1410a830 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 430158490 0 0) 0x7efc141a6ae0 con 0x7efc1410e400 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc00003520 con 0x7efc1410e400 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7efc0c001bf0 con 0x7efc1410e400 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011720908 0 0) 0x7efc141a7ce0 con 0x7efc14107990 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc141a6ae0 con 0x7efc14107990 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3698552031 0 0) 0x7efc141a8f20 con 0x7efc1410a830 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 >> v1:172.21.15.88:6790/0 conn(0x7efc1410e400 legacy=0x7efc141a51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 >> v1:172.21.15.138:6789/0 conn(0x7efc14107990 legacy=0x7efc1410dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc141aa160 con 0x7efc1410a830 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc141a9150 con 0x7efc1410a830 2024-03-26T21:56:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efc141a9620 con 0x7efc1410a830 2024-03-26T21:56:48.256 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7efc04003ae0 con 0x7efc1410a830 2024-03-26T21:56:48.256 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.256+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7efc04005c80 con 0x7efc1410a830 2024-03-26T21:56:48.256 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.257+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbf80050b0 con 0x7efc1410a830 2024-03-26T21:56:48.256 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.257+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7efc04003650 con 0x7efc1410a830 2024-03-26T21:56:48.256 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.258+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(818..818 src has 268..818) ==== 7799+0+0 (unknown 205995521 0 0) 0x7efc0408a5d0 con 0x7efc1410a830 2024-03-26T21:56:48.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.262+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7efc04057340 con 0x7efc1410a830 2024-03-26T21:56:48.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:48.444+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12} v 0) -- 0x7efbf80053a0 con 0x7efc1410a830 2024-03-26T21:56:49.118 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.122+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]=0 pool '4328d745-f987-428c-ba08-f658a5eea7e4' created v819) ==== 176+0+0 (unknown 1017490108 0 0) 0x7efc0405aff0 con 0x7efc1410a830 2024-03-26T21:56:49.233 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.236+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12} v 0) -- 0x7efbf80058b0 con 0x7efc1410a830 2024-03-26T21:56:49.233 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.237+0000 7efc0a7fc640 1 -- 172.21.15.88:0/2568962121 <== mon.0 v1:172.21.15.88:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]=0 pool '4328d745-f987-428c-ba08-f658a5eea7e4' already exists v819) ==== 183+0+0 (unknown 1012649163 0 0) 0x7efc0405f8b0 con 0x7efc1410a830 2024-03-26T21:56:49.233 INFO:tasks.workunit.client.0.smithi088.stderr:pool '4328d745-f987-428c-ba08-f658a5eea7e4' already exists 2024-03-26T21:56:49.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.239+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 >> v1:172.21.15.88:6800/3939589528 conn(0x7efc0006f580 legacy=0x7efc00071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.239+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 >> v1:172.21.15.88:6789/0 conn(0x7efc1410a830 legacy=0x7efc141a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.239+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 shutdown_connections 2024-03-26T21:56:49.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.239+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 >> 172.21.15.88:0/2568962121 conn(0x7efc140fd430 msgr2=0x7efc14100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:49.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.239+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 shutdown_connections 2024-03-26T21:56:49.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.239+0000 7efc1b916640 1 -- 172.21.15.88:0/2568962121 wait complete. 2024-03-26T21:56:49.245 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool application enable 4328d745-f987-428c-ba08-f658a5eea7e4 rados 2024-03-26T21:56:49.329 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.332+0000 7f8594965640 1 -- 172.21.15.88:0/3244445037 >> v1:172.21.15.88:6789/0 conn(0x7f8590109740 legacy=0x7f859010bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.329 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.332+0000 7f8594965640 1 -- 172.21.15.88:0/3244445037 shutdown_connections 2024-03-26T21:56:49.329 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.332+0000 7f8594965640 1 -- 172.21.15.88:0/3244445037 >> 172.21.15.88:0/3244445037 conn(0x7f8590077560 msgr2=0x7f85900fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:49.329 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.332+0000 7f8594965640 1 -- 172.21.15.88:0/3244445037 shutdown_connections 2024-03-26T21:56:49.329 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.332+0000 7f8594965640 1 -- 172.21.15.88:0/3244445037 wait complete. 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f8594965640 1 Processor -- start 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f8594965640 1 -- start start 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f8594965640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85901a5960 con 0x7f8590109740 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f8594965640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85901a6ba0 con 0x7f859010d310 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f8594965640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85901a7de0 con 0x7f85901068a0 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f858e59c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f85901068a0 0x7f859010cbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42618/0 (socket says 172.21.15.88:42618) 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f858e59c640 1 -- 172.21.15.88:0/2560620133 learned_addr learned my addr 172.21.15.88:0/2560620133 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f8587fff640 1 --1- 172.21.15.88:0/2560620133 >> v1:172.21.15.88:6789/0 conn(0x7f8590109740 0x7f85901a0760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60646/0 (socket says 172.21.15.88:60646) 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4239800519 0 0) 0x7f85901a7de0 con 0x7f85901068a0 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f857c003160 con 0x7f85901068a0 2024-03-26T21:56:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280235474 0 0) 0x7f85901a5960 con 0x7f8590109740 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.333+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85901a7de0 con 0x7f8590109740 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186692381 0 0) 0x7f85901a6ba0 con 0x7f859010d310 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85901a5960 con 0x7f859010d310 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3208247476 0 0) 0x7f857c003160 con 0x7f85901068a0 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85901a6ba0 con 0x7f85901068a0 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f85800018b0 con 0x7f85901068a0 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2596151672 0 0) 0x7f85901a7de0 con 0x7f8590109740 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f857c003160 con 0x7f8590109740 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f8578004320 con 0x7f8590109740 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1524212427 0 0) 0x7f85901a6ba0 con 0x7f85901068a0 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 >> v1:172.21.15.138:6789/0 conn(0x7f859010d310 legacy=0x7f85901a4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 >> v1:172.21.15.88:6789/0 conn(0x7f8590109740 legacy=0x7f85901a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85901a9020 con 0x7f85901068a0 2024-03-26T21:56:49.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85901a8010 con 0x7f85901068a0 2024-03-26T21:56:49.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f85901a84e0 con 0x7f85901068a0 2024-03-26T21:56:49.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f8580003aa0 con 0x7f85901068a0 2024-03-26T21:56:49.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.334+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f8580005ba0 con 0x7f85901068a0 2024-03-26T21:56:49.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.335+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8570005080 con 0x7f85901068a0 2024-03-26T21:56:49.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.335+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f8580003610 con 0x7f85901068a0 2024-03-26T21:56:49.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.336+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(819..819 src has 268..819) ==== 8174+0+0 (unknown 3848798355 0 0) 0x7f858008a540 con 0x7f85901068a0 2024-03-26T21:56:49.337 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.340+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f8580057140 con 0x7f85901068a0 2024-03-26T21:56:49.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:56:49.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:49 smithi138 ceph-mon[33706]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T21:56:49.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:49 smithi138 ceph-mon[33706]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:49.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:49 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]: dispatch 2024-03-26T21:56:49.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:56:49.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[39202]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T21:56:49.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[39202]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:49.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]: dispatch 2024-03-26T21:56:49.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/3245936251' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:56:49.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[30075]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T21:56:49.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[30075]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:56:49.490 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:49 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]: dispatch 2024-03-26T21:56:49.515 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:49.519+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"} v 0) -- 0x7f8570005370 con 0x7f85901068a0 2024-03-26T21:56:50.132 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:50.135+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]=0 enabled application 'rados' on pool '4328d745-f987-428c-ba08-f658a5eea7e4' v820) ==== 213+0+0 (unknown 2082856443 0 0) 0x7f858005adf0 con 0x7f85901068a0 2024-03-26T21:56:50.250 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:50.254+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"} v 0) -- 0x7f8570001090 con 0x7f85901068a0 2024-03-26T21:56:50.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]': finished 2024-03-26T21:56:50.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:50 smithi138 ceph-mon[33706]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T21:56:50.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]: dispatch 2024-03-26T21:56:50.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:50 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2560620133' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:50.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:50 smithi138 ceph-mon[33706]: from='client.57434 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:50.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:50 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:56:50.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]': finished 2024-03-26T21:56:50.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[30075]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T21:56:50.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]: dispatch 2024-03-26T21:56:50.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2560620133' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[30075]: from='client.57434 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]': finished 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[39202]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2568962121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pg_num": 12}]: dispatch 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2560620133' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[39202]: from='client.57434 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:50.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:50 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:56:51.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.142+0000 7f85877fe640 1 -- 172.21.15.88:0/2560620133 <== mon.2 v1:172.21.15.88:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]=0 enabled application 'rados' on pool '4328d745-f987-428c-ba08-f658a5eea7e4' v821) ==== 213+0+0 (unknown 1174678301 0 0) 0x7f858005f6b0 con 0x7f85901068a0 2024-03-26T21:56:51.139 INFO:tasks.workunit.client.0.smithi088.stderr:enabled application 'rados' on pool '4328d745-f987-428c-ba08-f658a5eea7e4' 2024-03-26T21:56:51.141 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.144+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 >> v1:172.21.15.88:6800/3939589528 conn(0x7f857c06f310 legacy=0x7f857c0717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:51.141 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.145+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 >> v1:172.21.15.88:6790/0 conn(0x7f85901068a0 legacy=0x7f859010cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:51.141 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.145+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 shutdown_connections 2024-03-26T21:56:51.142 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.145+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 >> 172.21.15.88:0/2560620133 conn(0x7f8590077560 msgr2=0x7f859010d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:51.142 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.145+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 shutdown_connections 2024-03-26T21:56:51.142 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.145+0000 7f8594965640 1 -- 172.21.15.88:0/2560620133 wait complete. 2024-03-26T21:56:51.150 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 4328d745-f987-428c-ba08-f658a5eea7e4 max_objects 10 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.235+0000 7f6f3b12b640 1 -- 172.21.15.88:0/633544004 >> v1:172.21.15.88:6790/0 conn(0x7f6f341068b0 legacy=0x7f6f34106c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.235+0000 7f6f3b12b640 1 -- 172.21.15.88:0/633544004 shutdown_connections 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.235+0000 7f6f3b12b640 1 -- 172.21.15.88:0/633544004 >> 172.21.15.88:0/633544004 conn(0x7f6f340fc370 msgr2=0x7f6f340fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.235+0000 7f6f3b12b640 1 -- 172.21.15.88:0/633544004 shutdown_connections 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.235+0000 7f6f3b12b640 1 -- 172.21.15.88:0/633544004 wait complete. 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f3b12b640 1 Processor -- start 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f3b12b640 1 -- start start 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f3b12b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f3410cfe0 con 0x7f6f34109750 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f3b12b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f341a7ee0 con 0x7f6f3410d320 2024-03-26T21:56:51.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f3b12b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f341a90c0 con 0x7f6f341068b0 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f38ec7640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f6f341068b0 0x7f6f34069ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42634/0 (socket says 172.21.15.88:42634) 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f33fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6f34109750 0x7f6f3406a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60668/0 (socket says 172.21.15.88:60668) 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f38ec7640 1 -- 172.21.15.88:0/130467425 learned_addr learned my addr 172.21.15.88:0/130467425 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.236+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3129783986 0 0) 0x7f6f3410cfe0 con 0x7f6f34109750 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f24003520 con 0x7f6f34109750 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090020853 0 0) 0x7f6f341a90c0 con 0x7f6f341068b0 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f3410cfe0 con 0x7f6f341068b0 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 920154840 0 0) 0x7f6f341a7ee0 con 0x7f6f3410d320 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f341a90c0 con 0x7f6f3410d320 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4018669998 0 0) 0x7f6f24003520 con 0x7f6f34109750 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f341a7ee0 con 0x7f6f34109750 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6f20001840 con 0x7f6f34109750 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4003035589 0 0) 0x7f6f3410cfe0 con 0x7f6f341068b0 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f24003520 con 0x7f6f341068b0 2024-03-26T21:56:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4041757061 0 0) 0x7f6f341a90c0 con 0x7f6f3410d320 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f3410cfe0 con 0x7f6f3410d320 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6f280043d0 con 0x7f6f341068b0 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6f2c001b70 con 0x7f6f3410d320 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3535817061 0 0) 0x7f6f341a7ee0 con 0x7f6f34109750 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 >> v1:172.21.15.88:6790/0 conn(0x7f6f341068b0 legacy=0x7f6f34069ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 >> v1:172.21.15.138:6789/0 conn(0x7f6f3410d320 legacy=0x7f6f341a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f341aa2a0 con 0x7f6f34109750 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f341a6f30 con 0x7f6f34109750 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f6f20003a80 con 0x7f6f34109750 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f6f20005ae0 con 0x7f6f34109750 2024-03-26T21:56:51.237 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.237+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f341a7400 con 0x7f6f34109750 2024-03-26T21:56:51.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.238+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f34101fe0 con 0x7f6f34109750 2024-03-26T21:56:51.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.239+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f6f200035f0 con 0x7f6f34109750 2024-03-26T21:56:51.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.239+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(821..821 src has 268..821) ==== 8187+0+0 (unknown 2846839952 0 0) 0x7f6f2008a5c0 con 0x7f6f34109750 2024-03-26T21:56:51.240 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.244+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f6f200571f0 con 0x7f6f34109750 2024-03-26T21:56:51.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:51.424+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"} v 0) -- 0x7f6f3410c2e0 con 0x7f6f34109750 2024-03-26T21:56:51.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: from='client.57434 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]': finished 2024-03-26T21:56:51.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T21:56:51.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: pgmap v1792: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:56:51.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2560620133' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:51.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: from='client.57434 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:51.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:56:51.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:56:51.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:51 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:56:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: from='client.57434 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]': finished 2024-03-26T21:56:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: pgmap v1792: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2560620133' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: from='client.57434 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: from='client.57434 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]': finished 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T21:56:51.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: pgmap v1792: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 863 MiB used, 714 GiB / 715 GiB avail 2024-03-26T21:56:51.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2560620133' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:51.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: from='client.57434 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]: dispatch 2024-03-26T21:56:51.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:56:51.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:56:51.488 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:51 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:56:52.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:52.143+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4328d745-f987-428c-ba08-f658a5eea7e4 v822) ==== 223+0+0 (unknown 336482614 0 0) 0x7f6f2005aea0 con 0x7f6f34109750 2024-03-26T21:56:52.250 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:52.253+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"} v 0) -- 0x7f6f341088a0 con 0x7f6f34109750 2024-03-26T21:56:52.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:52 smithi088 ceph-mon[39202]: from='client.57434 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]': finished 2024-03-26T21:56:52.469 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:52 smithi088 ceph-mon[39202]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T21:56:52.470 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:52 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:56:52.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:52 smithi088 ceph-mon[30075]: from='client.57434 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]': finished 2024-03-26T21:56:52.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:52 smithi088 ceph-mon[30075]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T21:56:52.470 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:52 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:56:52.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:56:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:56:52.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:52 smithi138 ceph-mon[33706]: from='client.57434 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "app": "rados"}]': finished 2024-03-26T21:56:52.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:52 smithi138 ceph-mon[33706]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T21:56:52.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:52 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:56:52.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:56:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:56:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:56:53.145 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:53.149+0000 7f6f31ffb640 1 -- 172.21.15.88:0/130467425 <== mon.0 v1:172.21.15.88:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4328d745-f987-428c-ba08-f658a5eea7e4 v823) ==== 223+0+0 (unknown 1565258822 0 0) 0x7f6f2005f760 con 0x7f6f34109750 2024-03-26T21:56:53.145 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_objects = 10 for pool 4328d745-f987-428c-ba08-f658a5eea7e4 2024-03-26T21:56:53.149 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:53.153+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 >> v1:172.21.15.88:6800/3939589528 conn(0x7f6f2406f680 legacy=0x7f6f24071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:53.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:53.153+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 >> v1:172.21.15.88:6789/0 conn(0x7f6f34109750 legacy=0x7f6f3406a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:56:53.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:53.154+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 shutdown_connections 2024-03-26T21:56:53.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:53.154+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 >> 172.21.15.88:0/130467425 conn(0x7f6f340fc370 msgr2=0x7f6f34110800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:56:53.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:53.154+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 shutdown_connections 2024-03-26T21:56:53.151 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:56:53.154+0000 7f6f3b12b640 1 -- 172.21.15.88:0/130467425 wait complete. 2024-03-26T21:56:53.162 INFO:tasks.workunit.client.0.smithi088.stderr:+ sleep 30 2024-03-26T21:56:53.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:56:53.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:53 smithi138 ceph-mon[33706]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T21:56:53.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:53 smithi138 ceph-mon[33706]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:56:53.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:53 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:56:53.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:53.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:56:53.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[30075]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[30075]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[39202]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[39202]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T21:56:53.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:56:54.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:56:54.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:54 smithi138 ceph-mon[33706]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T21:56:54.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:56:54.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:54 smithi088 ceph-mon[39202]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T21:56:54.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/130467425' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "field": "max_objects", "val": "10"}]': finished 2024-03-26T21:56:54.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:54 smithi088 ceph-mon[30075]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T21:56:55.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:55 smithi138 ceph-mon[33706]: pgmap v1797: 188 pgs: 188 active+clean; 744 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:56:55.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:55 smithi088 ceph-mon[39202]: pgmap v1797: 188 pgs: 188 active+clean; 744 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:56:55.486 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:55 smithi088 ceph-mon[30075]: pgmap v1797: 188 pgs: 188 active+clean; 744 KiB data, 863 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T21:56:57.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:57 smithi138 ceph-mon[33706]: pgmap v1798: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T21:56:57.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:57 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:56:57.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:57.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:57 smithi088 ceph-mon[39202]: pgmap v1798: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T21:56:57.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:57 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:56:57.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:57.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:57 smithi088 ceph-mon[30075]: pgmap v1798: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T21:56:57.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:57 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:56:57.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:56:59.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:56:59 smithi138 ceph-mon[33706]: pgmap v1799: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-26T21:56:59.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:56:59 smithi088 ceph-mon[39202]: pgmap v1799: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-26T21:56:59.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:56:59 smithi088 ceph-mon[30075]: pgmap v1799: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-26T21:57:01.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:01 smithi138 ceph-mon[33706]: pgmap v1800: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-26T21:57:01.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:01 smithi088 ceph-mon[39202]: pgmap v1800: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-26T21:57:01.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:01 smithi088 ceph-mon[30075]: pgmap v1800: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-26T21:57:02.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:57:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:57:02.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:57:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:57:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:57:03.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:03 smithi138 ceph-mon[33706]: pgmap v1801: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:57:03.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:03.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:03 smithi088 ceph-mon[39202]: pgmap v1801: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:57:03.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:03.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:03 smithi088 ceph-mon[30075]: pgmap v1801: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:57:03.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:05.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:05 smithi138 ceph-mon[33706]: pgmap v1802: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-03-26T21:57:05.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:05 smithi088 ceph-mon[39202]: pgmap v1802: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-03-26T21:57:05.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:05 smithi088 ceph-mon[30075]: pgmap v1802: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-03-26T21:57:07.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:07 smithi138 ceph-mon[33706]: pgmap v1803: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:07.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:07 smithi088 ceph-mon[39202]: pgmap v1803: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:07.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:07 smithi088 ceph-mon[30075]: pgmap v1803: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:09.477 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:09 smithi138 ceph-mon[33706]: pgmap v1804: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:09.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:09 smithi088 ceph-mon[30075]: pgmap v1804: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:09.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:09 smithi088 ceph-mon[39202]: pgmap v1804: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:11.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:11 smithi088 ceph-mon[30075]: pgmap v1805: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:11.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:11 smithi088 ceph-mon[39202]: pgmap v1805: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:11.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:11 smithi138 ceph-mon[33706]: pgmap v1805: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:12.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:12.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:57:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:57:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:12.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:12.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:57:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:57:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:57:13.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:13 smithi138 ceph-mon[33706]: pgmap v1806: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:13.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:13.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:13 smithi088 ceph-mon[30075]: pgmap v1806: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:13.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:13.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:13 smithi088 ceph-mon[39202]: pgmap v1806: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:13.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:15.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:15 smithi138 ceph-mon[33706]: pgmap v1807: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:15.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:15 smithi088 ceph-mon[30075]: pgmap v1807: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:15.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:15 smithi088 ceph-mon[39202]: pgmap v1807: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:17.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:17 smithi138 ceph-mon[33706]: pgmap v1808: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:17.736 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:17 smithi088 ceph-mon[30075]: pgmap v1808: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:17.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:17 smithi088 ceph-mon[39202]: pgmap v1808: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:19.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:19 smithi138 ceph-mon[33706]: pgmap v1809: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:19.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:19 smithi088 ceph-mon[30075]: pgmap v1809: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:19.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:19 smithi088 ceph-mon[39202]: pgmap v1809: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:21.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:21 smithi138 ceph-mon[33706]: pgmap v1810: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:21.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:21 smithi088 ceph-mon[39202]: pgmap v1810: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:21.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:21 smithi088 ceph-mon[30075]: pgmap v1810: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:22.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:57:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:57:22.736 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:57:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:57:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:57:23.164 INFO:tasks.workunit.client.0.smithi088.stderr:++ seq 1 10 2024-03-26T21:57:23.165 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.165 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj1 /etc/passwd 2024-03-26T21:57:23.204 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.204 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj2 /etc/passwd 2024-03-26T21:57:23.240 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.240 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj3 /etc/passwd 2024-03-26T21:57:23.274 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.274 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj4 /etc/passwd 2024-03-26T21:57:23.309 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.309 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj5 /etc/passwd 2024-03-26T21:57:23.342 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.342 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj6 /etc/passwd 2024-03-26T21:57:23.375 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.375 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj7 /etc/passwd 2024-03-26T21:57:23.407 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.407 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj8 /etc/passwd 2024-03-26T21:57:23.439 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.439 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj9 /etc/passwd 2024-03-26T21:57:23.472 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in `seq 1 10` 2024-03-26T21:57:23.472 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 4328d745-f987-428c-ba08-f658a5eea7e4 put obj10 /etc/passwd 2024-03-26T21:57:23.505 INFO:tasks.workunit.client.0.smithi088.stderr:+ sleep 30 2024-03-26T21:57:23.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:23 smithi138 ceph-mon[33706]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:23.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:23.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:23 smithi088 ceph-mon[39202]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:23.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:23.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:23 smithi088 ceph-mon[30075]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:23.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:25.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:25 smithi138 ceph-mon[33706]: pgmap v1812: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:57:25.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:25 smithi088 ceph-mon[30075]: pgmap v1812: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:57:25.739 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:25 smithi088 ceph-mon[39202]: pgmap v1812: 188 pgs: 188 active+clean; 744 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:57:27.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:27 smithi138 ceph-mon[33706]: pgmap v1813: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T21:57:27.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:27.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:27 smithi138 ceph-mon[33706]: pool '4328d745-f987-428c-ba08-f658a5eea7e4' is full (reached quota's max_objects: 10) 2024-03-26T21:57:27.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:27 smithi138 ceph-mon[33706]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:57:27.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:27 smithi138 ceph-mon[33706]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[39202]: pgmap v1813: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[39202]: pool '4328d745-f987-428c-ba08-f658a5eea7e4' is full (reached quota's max_objects: 10) 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[39202]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[39202]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[30075]: pgmap v1813: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[30075]: pool '4328d745-f987-428c-ba08-f658a5eea7e4' is full (reached quota's max_objects: 10) 2024-03-26T21:57:27.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[30075]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T21:57:27.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:27 smithi088 ceph-mon[30075]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T21:57:29.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:29 smithi138 ceph-mon[33706]: pgmap v1815: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:29.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:29 smithi088 ceph-mon[30075]: pgmap v1815: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:29.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:29 smithi088 ceph-mon[39202]: pgmap v1815: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:31.529 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:31 smithi138 ceph-mon[33706]: pgmap v1816: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:31.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:31 smithi088 ceph-mon[30075]: pgmap v1816: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:31.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:31 smithi088 ceph-mon[39202]: pgmap v1816: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:32.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:57:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:57:32.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:57:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:57:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:57:33.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:33 smithi138 ceph-mon[33706]: pgmap v1817: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:33.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:33 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:33.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:33 smithi088 ceph-mon[30075]: pgmap v1817: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:33.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:33 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:33.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:33 smithi088 ceph-mon[39202]: pgmap v1817: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:33.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:33 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:35.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:35 smithi138 ceph-mon[33706]: pgmap v1818: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:35.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:35 smithi088 ceph-mon[30075]: pgmap v1818: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:35.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:35 smithi088 ceph-mon[39202]: pgmap v1818: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T21:57:37.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:37 smithi138 ceph-mon[33706]: pgmap v1819: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:57:37.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:37 smithi088 ceph-mon[30075]: pgmap v1819: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:57:37.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:37 smithi088 ceph-mon[39202]: pgmap v1819: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:57:39.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:39 smithi138 ceph-mon[33706]: pgmap v1820: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-03-26T21:57:39.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:39 smithi088 ceph-mon[39202]: pgmap v1820: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-03-26T21:57:39.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:39 smithi088 ceph-mon[30075]: pgmap v1820: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-03-26T21:57:41.539 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:41 smithi088 ceph-mon[39202]: pgmap v1821: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:41.540 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:41 smithi088 ceph-mon[30075]: pgmap v1821: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:41.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:41 smithi138 ceph-mon[33706]: pgmap v1821: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:42.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:57:42 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:57:42.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:57:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:57:43.227 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:42 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:43.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:42 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:43.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:42 smithi138 ceph-mon[33706]: pgmap v1822: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:43.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:42 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[39202]: pgmap v1822: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[30075]: pgmap v1822: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:43.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:42 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:45.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:45 smithi088 ceph-mon[39202]: pgmap v1823: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:45.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:45 smithi088 ceph-mon[30075]: pgmap v1823: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:45.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:45 smithi138 ceph-mon[33706]: pgmap v1823: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:47.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:47 smithi088 ceph-mon[39202]: pgmap v1824: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:47.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:47 smithi088 ceph-mon[30075]: pgmap v1824: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:47.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:47 smithi138 ceph-mon[33706]: pgmap v1824: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:49.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:49 smithi088 ceph-mon[39202]: pgmap v1825: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:49.487 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:49 smithi088 ceph-mon[30075]: pgmap v1825: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:49.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:49 smithi138 ceph-mon[33706]: pgmap v1825: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:51.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:51 smithi138 ceph-mon[33706]: pgmap v1826: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:51.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:51 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:57:51.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:51 smithi088 ceph-mon[39202]: pgmap v1826: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:51.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:51 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:57:51.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:51 smithi088 ceph-mon[30075]: pgmap v1826: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:51.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:51 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:57:52.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:57:52 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:57:52.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:52 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:52 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:52 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:57:52.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:52 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:57:52.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:52 smithi138 ceph-mon[33706]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:57:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:57:52.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:57:52.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:57:52.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[39202]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:52.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:57:52.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:57:52.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:52 smithi088 ceph-mon[30075]: from='mgr.14622 ' entity='mgr.y' 2024-03-26T21:57:53.506 INFO:tasks.workunit.client.0.smithi088.stderr:+ rados -p 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 put threemore /etc/passwd 2024-03-26T21:57:53.545 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 max_bytes 0 2024-03-26T21:57:53.632 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.635+0000 7f496f802640 1 -- 172.21.15.88:0/3927404187 >> v1:172.21.15.88:6790/0 conn(0x7f49681060b0 legacy=0x7f4968106490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:53.632 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.635+0000 7f496f802640 1 -- 172.21.15.88:0/3927404187 shutdown_connections 2024-03-26T21:57:53.632 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.635+0000 7f496f802640 1 -- 172.21.15.88:0/3927404187 >> 172.21.15.88:0/3927404187 conn(0x7f4968075b80 msgr2=0x7f4968075f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:53.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.635+0000 7f496f802640 1 -- 172.21.15.88:0/3927404187 shutdown_connections 2024-03-26T21:57:53.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.635+0000 7f496f802640 1 -- 172.21.15.88:0/3927404187 wait complete. 2024-03-26T21:57:53.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f496f802640 1 Processor -- start 2024-03-26T21:57:53.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f496f802640 1 -- start start 2024-03-26T21:57:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f496f802640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4968104e00 con 0x7f49681060b0 2024-03-26T21:57:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f496f802640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49681a7e20 con 0x7f496810ca90 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f496f802640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49681a9040 con 0x7f4968108ec0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f496d59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f49681060b0 0x7f4968104230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41752/0 (socket says 172.21.15.88:41752) 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f496d59e640 1 -- 172.21.15.88:0/1710683291 learned_addr learned my addr 172.21.15.88:0/1710683291 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861034196 0 0) 0x7f4968104e00 con 0x7f49681060b0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4954004630 con 0x7f49681060b0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 378314154 0 0) 0x7f49681a9040 con 0x7f4968108ec0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4968104e00 con 0x7f4968108ec0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031341026 0 0) 0x7f49681a7e20 con 0x7f496810ca90 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49681a9040 con 0x7f496810ca90 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4080869422 0 0) 0x7f4954004630 con 0x7f49681060b0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49681a7e20 con 0x7f49681060b0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4964001950 con 0x7f49681060b0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.636+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 497974950 0 0) 0x7f4968104e00 con 0x7f4968108ec0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4954004630 con 0x7f4968108ec0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4958001720 con 0x7f4968108ec0 2024-03-26T21:57:53.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2285389671 0 0) 0x7f49681a9040 con 0x7f496810ca90 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4968104e00 con 0x7f496810ca90 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f49600018b0 con 0x7f496810ca90 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1522591347 0 0) 0x7f49681a7e20 con 0x7f49681060b0 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 >> v1:172.21.15.88:6790/0 conn(0x7f4968108ec0 legacy=0x7f49681a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 >> v1:172.21.15.138:6789/0 conn(0x7f496810ca90 legacy=0x7f49681a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49681aa2a0 con 0x7f49681060b0 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49681a9270 con 0x7f49681060b0 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f49681a9740 con 0x7f49681060b0 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f4964003970 con 0x7f49681060b0 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.637+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f4964005020 con 0x7f49681060b0 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.638+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f494c005080 con 0x7f49681060b0 2024-03-26T21:57:53.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.638+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f4964005000 con 0x7f49681060b0 2024-03-26T21:57:53.638 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.639+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(824..824 src has 268..824) ==== 8187+0+0 (unknown 2520940318 0 0) 0x7f496408a880 con 0x7f49681060b0 2024-03-26T21:57:53.638 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.639+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=825}) -- 0x7f49681a7e20 con 0x7f49681060b0 2024-03-26T21:57:53.641 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.644+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f49640575d0 con 0x7f49681060b0 2024-03-26T21:57:53.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:53 smithi138 ceph-mon[33706]: pgmap v1827: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:53.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:53 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:53.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:53 smithi088 ceph-mon[39202]: pgmap v1827: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:53.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:53 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:53.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:53 smithi088 ceph-mon[30075]: pgmap v1827: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:57:53.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:53 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:57:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:53.822+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"} v 0) -- 0x7f494c005370 con 0x7f49681060b0 2024-03-26T21:57:54.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:54.319+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v825) ==== 217+0+0 (unknown 189928554 0 0) 0x7f496405b280 con 0x7f49681060b0 2024-03-26T21:57:54.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:54.319+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 11 ==== osd_map(825..825 src has 268..825) ==== 628+0+0 (unknown 3477290018 0 0) 0x7f49640534b0 con 0x7f49681060b0 2024-03-26T21:57:54.317 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:54.320+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=826}) -- 0x7f4954004630 con 0x7f49681060b0 2024-03-26T21:57:54.429 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:54.433+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"} v 0) -- 0x7f494c000c40 con 0x7f49681060b0 2024-03-26T21:57:54.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:54 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:57:54.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:54 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:57:54.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:54 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:57:55.318 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.321+0000 7f495e7fc640 1 -- 172.21.15.88:0/1710683291 <== mon.0 v1:172.21.15.88:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v826) ==== 217+0+0 (unknown 2102621385 0 0) 0x7f496405fb40 con 0x7f49681060b0 2024-03-26T21:57:55.318 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_bytes = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:57:55.322 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.325+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 >> v1:172.21.15.88:6800/3939589528 conn(0x7f495406f530 legacy=0x7f49540719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:55.322 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.325+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 >> v1:172.21.15.88:6789/0 conn(0x7f49681060b0 legacy=0x7f4968104230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:55.322 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.325+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 shutdown_connections 2024-03-26T21:57:55.322 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.325+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 >> 172.21.15.88:0/1710683291 conn(0x7f4968075b80 msgr2=0x7f4968107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:55.323 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.325+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 shutdown_connections 2024-03-26T21:57:55.323 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.325+0000 7f496f802640 1 -- 172.21.15.88:0/1710683291 wait complete. 2024-03-26T21:57:55.336 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool set-quota 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 max_objects 0 2024-03-26T21:57:55.420 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.423+0000 7fa936513640 1 -- 172.21.15.88:0/3497590154 >> v1:172.21.15.88:6790/0 conn(0x7fa93010d310 legacy=0x7fa93010f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.423+0000 7fa936513640 1 -- 172.21.15.88:0/3497590154 shutdown_connections 2024-03-26T21:57:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.423+0000 7fa936513640 1 -- 172.21.15.88:0/3497590154 >> 172.21.15.88:0/3497590154 conn(0x7fa930077560 msgr2=0x7fa9300fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.423+0000 7fa936513640 1 -- 172.21.15.88:0/3497590154 shutdown_connections 2024-03-26T21:57:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.423+0000 7fa936513640 1 -- 172.21.15.88:0/3497590154 wait complete. 2024-03-26T21:57:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa936513640 1 Processor -- start 2024-03-26T21:57:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa936513640 1 -- start start 2024-03-26T21:57:55.423 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa936513640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa93010cf00 con 0x7fa9301068a0 2024-03-26T21:57:55.423 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa936513640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa93010d130 con 0x7fa930109740 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa936513640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9301a8f40 con 0x7fa93010d310 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa92ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa9301068a0 0x7fa93010ad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41768/0 (socket says 172.21.15.88:41768) 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa92ffff640 1 -- 172.21.15.88:0/747929458 learned_addr learned my addr 172.21.15.88:0/747929458 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 939078570 0 0) 0x7fa9301a8f40 con 0x7fa93010d310 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.424+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa918003160 con 0x7fa93010d310 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497663826 0 0) 0x7fa93010cf00 con 0x7fa9301068a0 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9301a8f40 con 0x7fa9301068a0 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3122380656 0 0) 0x7fa9301122f0 con 0x7fa930109740 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa93010cf00 con 0x7fa930109740 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3259038452 0 0) 0x7fa918003160 con 0x7fa93010d310 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9301122f0 con 0x7fa93010d310 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fa928001bd0 con 0x7fa93010d310 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1963709333 0 0) 0x7fa9301a8f40 con 0x7fa9301068a0 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa918003160 con 0x7fa9301068a0 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fa920001230 con 0x7fa9301068a0 2024-03-26T21:57:55.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1597869515 0 0) 0x7fa93010cf00 con 0x7fa930109740 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9301a8f40 con 0x7fa930109740 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fa924001830 con 0x7fa930109740 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 129641104 0 0) 0x7fa9301122f0 con 0x7fa93010d310 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 >> v1:172.21.15.138:6789/0 conn(0x7fa930109740 legacy=0x7fa93010b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 >> v1:172.21.15.88:6789/0 conn(0x7fa9301068a0 legacy=0x7fa93010ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9301aa180 con 0x7fa93010d310 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9301a9170 con 0x7fa93010d310 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9301a9640 con 0x7fa93010d310 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7fa928001e90 con 0x7fa93010d310 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.425+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7fa928004f60 con 0x7fa93010d310 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.426+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa91c005080 con 0x7fa93010d310 2024-03-26T21:57:55.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.426+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7fa928003970 con 0x7fa93010d310 2024-03-26T21:57:55.426 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.427+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(826..826 src has 268..826) ==== 8187+0+0 (unknown 944648849 0 0) 0x7fa928089730 con 0x7fa93010d310 2024-03-26T21:57:55.426 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.427+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=827}) -- 0x7fa9301122f0 con 0x7fa93010d310 2024-03-26T21:57:55.429 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.432+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7fa928056360 con 0x7fa93010d310 2024-03-26T21:57:55.606 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:55.609+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"} v 0) -- 0x7fa91c005370 con 0x7fa93010d310 2024-03-26T21:57:55.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:55 smithi138 ceph-mon[33706]: pgmap v1828: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:55.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:57:55.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:55 smithi138 ceph-mon[33706]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T21:57:55.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:55 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[30075]: pgmap v1828: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[30075]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[39202]: pgmap v1828: 188 pgs: 188 active+clean; 859 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[39202]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T21:57:55.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:55 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T21:57:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:56.343+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 10 ==== osd_map(827..827 src has 268..827) ==== 628+0+0 (unknown 844512248 0 0) 0x7fa9280873a0 con 0x7fa93010d310 2024-03-26T21:57:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:56.343+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=828}) -- 0x7fa93010cf00 con 0x7fa93010d310 2024-03-26T21:57:56.348 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:56.351+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v827) ==== 221+0+0 (unknown 847262884 0 0) 0x7fa92805a010 con 0x7fa93010d310 2024-03-26T21:57:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:56.458+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"} v 0) -- 0x7fa91c000c40 con 0x7fa93010d310 2024-03-26T21:57:56.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:57:56.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:56 smithi138 ceph-mon[33706]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T21:57:56.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:56 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/747929458' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:56.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:56 smithi138 ceph-mon[33706]: from='client.57551 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:56.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:57:56.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[30075]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T21:57:56.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/747929458' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:56.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[30075]: from='client.57551 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:56.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1710683291' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T21:57:56.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[39202]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T21:57:56.738 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/747929458' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:56.738 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:56 smithi088 ceph-mon[39202]: from='client.57551 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:57.342 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.345+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 12 ==== osd_map(828..828 src has 268..828) ==== 628+0+0 (unknown 479045318 0 0) 0x7fa928051ca0 con 0x7fa93010d310 2024-03-26T21:57:57.342 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.345+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=829}) -- 0x7fa91807cff0 con 0x7fa93010d310 2024-03-26T21:57:57.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.349+0000 7fa92d7fa640 1 -- 172.21.15.88:0/747929458 <== mon.2 v1:172.21.15.88:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 v828) ==== 221+0+0 (unknown 3171326480 0 0) 0x7fa92805e8d0 con 0x7fa93010d310 2024-03-26T21:57:57.346 INFO:tasks.workunit.client.0.smithi088.stderr:set-quota max_objects = 0 for pool 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 2024-03-26T21:57:57.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.353+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 >> v1:172.21.15.88:6800/3939589528 conn(0x7fa91806f640 legacy=0x7fa918071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:57.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.353+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 >> v1:172.21.15.88:6790/0 conn(0x7fa93010d310 legacy=0x7fa9301a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:57:57.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.353+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 shutdown_connections 2024-03-26T21:57:57.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.353+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 >> 172.21.15.88:0/747929458 conn(0x7fa930077560 msgr2=0x7fa930110810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:57:57.351 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.353+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 shutdown_connections 2024-03-26T21:57:57.351 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:57:57.353+0000 7fa936513640 1 -- 172.21.15.88:0/747929458 wait complete. 2024-03-26T21:57:57.362 INFO:tasks.workunit.client.0.smithi088.stderr:+ sleep 30 2024-03-26T21:57:57.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:57 smithi138 ceph-mon[33706]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T21:57:57.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:57 smithi138 ceph-mon[33706]: from='client.57551 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:57:57.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:57 smithi138 ceph-mon[33706]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T21:57:57.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:57 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/747929458' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:57.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:57 smithi138 ceph-mon[33706]: from='client.57551 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:57.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:57 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[30075]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[30075]: from='client.57551 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[30075]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/747929458' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[30075]: from='client.57551 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[39202]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[39202]: from='client.57551 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:57:57.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[39202]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T21:57:57.738 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/747929458' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:57.738 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[39202]: from='client.57551 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T21:57:57.738 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:57 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:57:58.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:58 smithi138 ceph-mon[33706]: from='client.57551 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:57:58.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:58 smithi138 ceph-mon[33706]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T21:57:58.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:58 smithi088 ceph-mon[30075]: from='client.57551 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:57:58.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:58 smithi088 ceph-mon[30075]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T21:57:58.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:58 smithi088 ceph-mon[39202]: from='client.57551 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "field": "max_objects", "val": "0"}]': finished 2024-03-26T21:57:58.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:58 smithi088 ceph-mon[39202]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T21:57:59.727 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:57:59 smithi138 ceph-mon[33706]: pgmap v1834: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T21:57:59.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:57:59 smithi088 ceph-mon[39202]: pgmap v1834: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T21:57:59.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:57:59 smithi088 ceph-mon[30075]: pgmap v1834: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T21:58:01.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:01 smithi138 ceph-mon[33706]: pgmap v1835: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 874 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T21:58:01.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:01 smithi088 ceph-mon[30075]: pgmap v1835: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 874 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T21:58:01.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:01 smithi088 ceph-mon[39202]: pgmap v1835: 188 pgs: 188 active+clean; 871 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 874 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T21:58:02.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:58:02 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:58:02.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:02 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:58:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:58:03.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:03 smithi138 ceph-mon[33706]: pgmap v1836: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-26T21:58:03.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:03 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:03.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:03 smithi088 ceph-mon[30075]: pgmap v1836: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-26T21:58:03.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:03 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:03.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:03 smithi088 ceph-mon[39202]: pgmap v1836: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-26T21:58:03.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:03 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:05.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:05 smithi138 ceph-mon[33706]: pgmap v1837: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:58:05.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:05 smithi088 ceph-mon[30075]: pgmap v1837: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:58:05.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:05 smithi088 ceph-mon[39202]: pgmap v1837: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:58:07.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:07 smithi138 ceph-mon[33706]: pgmap v1838: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:58:07.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:07 smithi088 ceph-mon[39202]: pgmap v1838: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:58:07.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:07 smithi088 ceph-mon[30075]: pgmap v1838: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:58:09.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:09 smithi138 ceph-mon[33706]: pgmap v1839: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-26T21:58:09.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:09 smithi088 ceph-mon[39202]: pgmap v1839: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-26T21:58:09.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:09 smithi088 ceph-mon[30075]: pgmap v1839: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-26T21:58:11.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:11 smithi138 ceph-mon[33706]: pgmap v1840: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:11.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:11 smithi088 ceph-mon[39202]: pgmap v1840: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:11.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:11 smithi088 ceph-mon[30075]: pgmap v1840: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:12.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:12 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:58:12.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:58:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:58:12.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:12 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:58:12.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:12 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:58:12.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:12 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:58:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:58:13.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:13 smithi138 ceph-mon[33706]: pgmap v1841: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:13.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:13 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:13.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:13 smithi088 ceph-mon[39202]: pgmap v1841: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:13.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:13 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:13.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:13 smithi088 ceph-mon[30075]: pgmap v1841: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:13.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:13 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:15.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:15 smithi138 ceph-mon[33706]: pgmap v1842: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:15.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:15 smithi088 ceph-mon[30075]: pgmap v1842: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:15.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:15 smithi088 ceph-mon[39202]: pgmap v1842: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:17.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:17 smithi138 ceph-mon[33706]: pgmap v1843: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:17.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:17 smithi088 ceph-mon[39202]: pgmap v1843: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:17.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:17 smithi088 ceph-mon[30075]: pgmap v1843: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:19.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:19 smithi138 ceph-mon[33706]: pgmap v1844: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:19.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:19 smithi088 ceph-mon[39202]: pgmap v1844: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:19.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:19 smithi088 ceph-mon[30075]: pgmap v1844: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:21.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:21 smithi138 ceph-mon[33706]: pgmap v1845: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:21.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:21 smithi088 ceph-mon[30075]: pgmap v1845: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:21.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:21 smithi088 ceph-mon[39202]: pgmap v1845: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:22.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:58:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:58:22.737 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:22 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:58:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:58:23.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:23 smithi138 ceph-mon[33706]: pgmap v1846: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:23.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:23 smithi138 ceph-mon[33706]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:23.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:23 smithi088 ceph-mon[39202]: pgmap v1846: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:23.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:23 smithi088 ceph-mon[39202]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:23.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:23 smithi088 ceph-mon[30075]: pgmap v1846: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:23.738 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:23 smithi088 ceph-mon[30075]: from='client.24463 v1:172.21.15.138:0/2687352291' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:58:25.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:25 smithi138 ceph-mon[33706]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:25.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:25 smithi088 ceph-mon[39202]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:25.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:25 smithi088 ceph-mon[30075]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:27.363 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool delete 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 3f33c5ed-a35f-4328-bc45-4a61cfb59f82 --yes-i-really-really-mean-it 2024-03-26T21:58:27.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.459+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/3727415135 >> v1:172.21.15.88:6789/0 conn(0x7f7a6410d300 legacy=0x7f7a6410f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:27.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.459+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/3727415135 shutdown_connections 2024-03-26T21:58:27.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.459+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/3727415135 >> 172.21.15.88:0/3727415135 conn(0x7f7a640fc350 msgr2=0x7f7a640fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:27.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.459+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/3727415135 shutdown_connections 2024-03-26T21:58:27.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.459+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/3727415135 wait complete. 2024-03-26T21:58:27.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a6a4f4640 1 Processor -- start 2024-03-26T21:58:27.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a6a4f4640 1 -- start start 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a6a4f4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a6410cf00 con 0x7f7a6410d300 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a6a4f4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a6410d130 con 0x7f7a64106890 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a6a4f4640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a641a8f30 con 0x7f7a64109730 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a68a91640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7a6410d300 0x7f7a641a5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41536/0 (socket says 172.21.15.88:41536) 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a68a91640 1 -- 172.21.15.88:0/2129177844 learned_addr learned my addr 172.21.15.88:0/2129177844 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.460+0000 7f7a637fe640 1 --1- 172.21.15.88:0/2129177844 >> v1:172.21.15.88:6790/0 conn(0x7f7a64109730 0x7f7a6410b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:56808/0 (socket says 172.21.15.88:56808) 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743040346 0 0) 0x7f7a6410cf00 con 0x7f7a6410d300 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a4c003520 con 0x7f7a6410d300 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3742433186 0 0) 0x7f7a641a8f30 con 0x7f7a64109730 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a6410cf00 con 0x7f7a64109730 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536095867 0 0) 0x7f7a6410d130 con 0x7f7a64106890 2024-03-26T21:58:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a641a8f30 con 0x7f7a64106890 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2204089346 0 0) 0x7f7a4c003520 con 0x7f7a6410d300 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a6410d130 con 0x7f7a6410d300 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7a5c004830 con 0x7f7a6410d300 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3072909815 0 0) 0x7f7a6410cf00 con 0x7f7a64109730 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a4c003520 con 0x7f7a64109730 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7a58001830 con 0x7f7a64109730 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1721693916 0 0) 0x7f7a641a8f30 con 0x7f7a64106890 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a6410cf00 con 0x7f7a64106890 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7a54001250 con 0x7f7a64106890 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2899817967 0 0) 0x7f7a6410d130 con 0x7f7a6410d300 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 >> v1:172.21.15.88:6790/0 conn(0x7f7a64109730 legacy=0x7f7a6410b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 >> v1:172.21.15.138:6789/0 conn(0x7f7a64106890 legacy=0x7f7a6410ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a641aa170 con 0x7f7a6410d300 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a641a9160 con 0x7f7a6410d300 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a641a9630 con 0x7f7a6410d300 2024-03-26T21:58:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f7a5c001440 con 0x7f7a6410d300 2024-03-26T21:58:27.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.461+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f7a5c004f50 con 0x7f7a6410d300 2024-03-26T21:58:27.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.462+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a50005080 con 0x7f7a6410d300 2024-03-26T21:58:27.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.463+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f7a5c001da0 con 0x7f7a6410d300 2024-03-26T21:58:27.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.463+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(828..828 src has 268..828) ==== 8187+0+0 (unknown 3484848881 0 0) 0x7f7a5c08a720 con 0x7f7a6410d300 2024-03-26T21:58:27.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.463+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=829}) -- 0x7f7a6410d130 con 0x7f7a6410d300 2024-03-26T21:58:27.465 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.468+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f7a5c057310 con 0x7f7a6410d300 2024-03-26T21:58:27.643 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:27.646+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7a50005370 con 0x7f7a6410d300 2024-03-26T21:58:27.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:27 smithi138 ceph-mon[33706]: pgmap v1848: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:27.729 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:27 smithi138 ceph-mon[33706]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:58:27.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:27 smithi088 ceph-mon[30075]: pgmap v1848: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:27.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:27 smithi088 ceph-mon[30075]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:58:27.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:27 smithi088 ceph-mon[39202]: pgmap v1848: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:58:27.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:27 smithi088 ceph-mon[39202]: from='mgr.14622 v1:172.21.15.88:0/2975626305' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:58:28.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.414+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]=0 pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' removed v829) ==== 248+0+0 (unknown 2238065915 0 0) 0x7f7a5c05afc0 con 0x7f7a6410d300 2024-03-26T21:58:28.414 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.417+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 11 ==== osd_map(829..829 src has 268..829) ==== 296+0+0 (unknown 3248271171 0 0) 0x7f7a5c0528d0 con 0x7f7a6410d300 2024-03-26T21:58:28.414 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.417+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=830}) -- 0x7f7a4c003520 con 0x7f7a6410d300 2024-03-26T21:58:28.529 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.532+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7a50005da0 con 0x7f7a6410d300 2024-03-26T21:58:28.529 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.532+0000 7f7a617fa640 1 -- 172.21.15.88:0/2129177844 <== mon.0 v1:172.21.15.88:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]=0 pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' does not exist v829) ==== 255+0+0 (unknown 2848550482 0 0) 0x7f7a5c05f880 con 0x7f7a6410d300 2024-03-26T21:58:28.529 INFO:tasks.workunit.client.0.smithi088.stderr:pool '3f33c5ed-a35f-4328-bc45-4a61cfb59f82' does not exist 2024-03-26T21:58:28.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.534+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 >> v1:172.21.15.88:6800/3939589528 conn(0x7f7a4c06f5d0 legacy=0x7f7a4c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:28.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.534+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 >> v1:172.21.15.88:6789/0 conn(0x7f7a6410d300 legacy=0x7f7a641a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:28.532 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.534+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 shutdown_connections 2024-03-26T21:58:28.532 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.534+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 >> 172.21.15.88:0/2129177844 conn(0x7f7a640fc350 msgr2=0x7f7a640ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:28.532 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.534+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 shutdown_connections 2024-03-26T21:58:28.532 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.534+0000 7f7a6a4f4640 1 -- 172.21.15.88:0/2129177844 wait complete. 2024-03-26T21:58:28.540 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph osd pool delete 4328d745-f987-428c-ba08-f658a5eea7e4 4328d745-f987-428c-ba08-f658a5eea7e4 --yes-i-really-really-mean-it 2024-03-26T21:58:28.622 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.624+0000 7f59acbeb640 1 -- 172.21.15.88:0/4183506585 >> v1:172.21.15.88:6790/0 conn(0x7f59a8106890 legacy=0x7f59a8106c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:28.622 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.625+0000 7f59acbeb640 1 -- 172.21.15.88:0/4183506585 shutdown_connections 2024-03-26T21:58:28.622 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.625+0000 7f59acbeb640 1 -- 172.21.15.88:0/4183506585 >> 172.21.15.88:0/4183506585 conn(0x7f59a80fc350 msgr2=0x7f59a80fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:28.622 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.625+0000 7f59acbeb640 1 -- 172.21.15.88:0/4183506585 shutdown_connections 2024-03-26T21:58:28.622 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.625+0000 7f59acbeb640 1 -- 172.21.15.88:0/4183506585 wait complete. 2024-03-26T21:58:28.625 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.625+0000 7f59acbeb640 1 Processor -- start 2024-03-26T21:58:28.625 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.625+0000 7f59acbeb640 1 -- start start 2024-03-26T21:58:28.625 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.625+0000 7f59acbeb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59a81a6970 con 0x7f59a8109730 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59acbeb640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59a81a7bb0 con 0x7f59a8106890 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59acbeb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59a81a8df0 con 0x7f59a810d300 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59a5d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f59a8109730 0x7f59a81a1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41548/0 (socket says 172.21.15.88:41548) 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59a6d9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f59a810d300 0x7f59a81a5070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:56830/0 (socket says 172.21.15.88:56830) 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59a5d9b640 1 -- 172.21.15.88:0/1711871315 learned_addr learned my addr 172.21.15.88:0/1711871315 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 539525502 0 0) 0x7f59a81a6970 con 0x7f59a8109730 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5990003160 con 0x7f59a8109730 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2237750611 0 0) 0x7f59a81a8df0 con 0x7f59a810d300 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a81a6970 con 0x7f59a810d300 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.1 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347166975 0 0) 0x7f59a81a7bb0 con 0x7f59a8106890 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a81a8df0 con 0x7f59a8106890 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 719912725 0 0) 0x7f5990003160 con 0x7f59a8109730 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59a81a7bb0 con 0x7f59a8109730 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f599c001a30 con 0x7f59a8109730 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 906743355 0 0) 0x7f59a81a6970 con 0x7f59a810d300 2024-03-26T21:58:28.626 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5990003160 con 0x7f59a810d300 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f59a0001af0 con 0x7f59a810d300 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.1 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 695827281 0 0) 0x7f59a81a8df0 con 0x7f59a8106890 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.138:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59a81a6970 con 0x7f59a8106890 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.1 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f5998004530 con 0x7f59a8106890 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3088273219 0 0) 0x7f59a81a7bb0 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 >> v1:172.21.15.88:6790/0 conn(0x7f59a810d300 legacy=0x7f59a81a5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 >> v1:172.21.15.138:6789/0 conn(0x7f59a8106890 legacy=0x7f59a810cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a81aa030 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59a81a6ba0 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f59a81a7070 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 3888493019 0 0) 0x7f599c001c90 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.626+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3671019322 0 0) 0x7f599c005bc0 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.627+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59880050b0 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.628+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 22) ==== 92496+0+0 (unknown 14161180 0 0) 0x7f599c0038a0 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.628+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(829..829 src has 268..829) ==== 7799+0+0 (unknown 1400108352 0 0) 0x7f599c08a2c0 con 0x7f59a8109730 2024-03-26T21:58:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.628+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=830}) -- 0x7f59a81a7bb0 con 0x7f59a8109730 2024-03-26T21:58:28.630 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.633+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187417 (unknown 1092875540 0 758779688) 0x7f599c057030 con 0x7f59a8109730 2024-03-26T21:58:28.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:28 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:28.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:28 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:28.737 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:28 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:28.806 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:28.809+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true} v 0) -- 0x7f59880053a0 con 0x7f59a8109730 2024-03-26T21:58:29.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.416+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]=0 pool '4328d745-f987-428c-ba08-f658a5eea7e4' removed v830) ==== 248+0+0 (unknown 3829007074 0 0) 0x7f599c05ace0 con 0x7f59a8109730 2024-03-26T21:58:29.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.416+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 11 ==== osd_map(830..830 src has 268..830) ==== 296+0+0 (unknown 3126313109 0 0) 0x7f599c088070 con 0x7f59a8109730 2024-03-26T21:58:29.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.416+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=831}) -- 0x7f59a81a8df0 con 0x7f59a8109730 2024-03-26T21:58:29.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.534+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5988005dd0 con 0x7f59a8109730 2024-03-26T21:58:29.531 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.534+0000 7f59977fe640 1 -- 172.21.15.88:0/1711871315 <== mon.0 v1:172.21.15.88:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]=0 pool '4328d745-f987-428c-ba08-f658a5eea7e4' does not exist v830) ==== 255+0+0 (unknown 2155257125 0 0) 0x7f599c05f5a0 con 0x7f59a8109730 2024-03-26T21:58:29.532 INFO:tasks.workunit.client.0.smithi088.stderr:pool '4328d745-f987-428c-ba08-f658a5eea7e4' does not exist 2024-03-26T21:58:29.534 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.536+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 >> v1:172.21.15.88:6800/3939589528 conn(0x7f599006f640 legacy=0x7f5990071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:29.534 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.536+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 >> v1:172.21.15.88:6789/0 conn(0x7f59a8109730 legacy=0x7f59a81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T21:58:29.534 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.536+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 shutdown_connections 2024-03-26T21:58:29.534 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.536+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 >> 172.21.15.88:0/1711871315 conn(0x7f59a80fc350 msgr2=0x7f59a8100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T21:58:29.534 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.536+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 shutdown_connections 2024-03-26T21:58:29.534 INFO:tasks.workunit.client.0.smithi088.stderr:2024-03-26T21:58:29.536+0000 7f59acbeb640 1 -- 172.21.15.88:0/1711871315 wait complete. 2024-03-26T21:58:29.542 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2024-03-26T21:58:29.543 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2024-03-26T21:58:29.544 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T21:58:29.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T21:58:29.612 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-26T21:58:29.612 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T21:58:29.676 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[39202]: pgmap v1849: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[39202]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[30075]: pgmap v1849: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[30075]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:29.677 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:29 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:29.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:29 smithi138 ceph-mon[33706]: pgmap v1849: 188 pgs: 188 active+clean; 871 KiB data, 865 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:58:29.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:58:29.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:29 smithi138 ceph-mon[33706]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T21:58:29.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/2129177844' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "pool2": "3f33c5ed-a35f-4328-bc45-4a61cfb59f82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:29.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:29 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:29.921 DEBUG:teuthology.parallel:result is None 2024-03-26T21:58:29.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T21:58:29.988 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T21:58:29.988 DEBUG:teuthology.orchestra.run.smithi088:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T21:58:30.048 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T21:58:30.048 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T21:58:30.058 INFO:tasks.cephadm:Teardown begin 2024-03-26T21:58:30.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:58:30.114 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:58:30.149 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T21:58:30.149 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T21:58:30.169 DEBUG:teuthology.orchestra.run.smithi138:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T21:58:30.207 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T21:58:30.207 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T21:58:30.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a 2024-03-26T21:58:30.519 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[39202]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:58:30.519 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:58:30.519 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[39202]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T21:58:30.520 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[39202]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:30.520 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 systemd[1]: Stopping Ceph mon.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:30.520 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[30075]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:58:30.520 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:58:30.520 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[30075]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T21:58:30.520 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-mon[30075]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:30.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:30 smithi138 ceph-mon[33706]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T21:58:30.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:58:30.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:30 smithi138 ceph-mon[33706]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T21:58:30.728 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:30 smithi138 ceph-mon[33706]: from='client.? v1:172.21.15.88:0/1711871315' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4328d745-f987-428c-ba08-f658a5eea7e4", "pool2": "4328d745-f987-428c-ba08-f658a5eea7e4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:58:30.859 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a[30052]: 2024-03-26T21:58:30.521+0000 7f119567a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:58:30.859 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a[30052]: 2024-03-26T21:58:30.521+0000 7f119567a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T21:58:30.859 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:30 smithi088 podman[157145]: 2024-03-26 21:58:30.563300502 +0000 UTC m=+0.165097571 container died e7a95e70ceca0ad07fa04ad6629e159bc0d7bae26810f5b3d677232cbe8acdce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=) 2024-03-26T21:58:31.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:31 smithi088 podman[157145]: 2024-03-26 21:58:31.019683671 +0000 UTC m=+0.621480817 container cleanup e7a95e70ceca0ad07fa04ad6629e159bc0d7bae26810f5b3d677232cbe8acdce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-26T21:58:31.237 INFO:journalctl@ceph.mon.a.smithi088.stdout:Mar 26 21:58:31 smithi088 bash[157145]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-a 2024-03-26T21:58:31.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T21:58:31.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:58:31.409 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T21:58:31.409 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T21:58:31.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.c 2024-03-26T21:58:31.934 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:31 smithi088 systemd[1]: Stopping Ceph mon.c for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:31.934 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:31 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-c[39179]: 2024-03-26T21:58:31.742+0000 7efc890f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:58:31.934 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:31 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-c[39179]: 2024-03-26T21:58:31.742+0000 7efc890f0700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T21:58:32.230 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:31 smithi088 podman[157275]: 2024-03-26 21:58:31.937668802 +0000 UTC m=+0.301527194 container died a36647f7c14f3a2241fcce62acf2983ac7f2512c1e3f5db49bd4172221161966 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:58:32.487 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:32 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: ::ffff:172.21.15.138 - - [26/Mar/2024:21:58:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:58:32.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:32 smithi088 podman[157275]: 2024-03-26 21:58:32.324063763 +0000 UTC m=+0.687922160 container cleanup a36647f7c14f3a2241fcce62acf2983ac7f2512c1e3f5db49bd4172221161966 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T21:58:32.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 26 21:58:32 smithi088 bash[157275]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-c 2024-03-26T21:58:32.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:58:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug there is no tcmu-runner data available 2024-03-26T21:58:32.735 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T21:58:32.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:58:32.772 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T21:58:32.772 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T21:58:32.772 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.b 2024-03-26T21:58:33.071 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:32 smithi138 systemd[1]: Stopping Ceph mon.b for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:33.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:33 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b[33670]: 2024-03-26T21:58:33.074+0000 7f75f4eb7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:58:33.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:33 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b[33670]: 2024-03-26T21:58:33.074+0000 7f75f4eb7700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T21:58:33.478 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:33 smithi138 podman[78594]: 2024-03-26 21:58:33.28971246 +0000 UTC m=+0.320967545 container died 985d4c5f9dcb5251e5704a968ce3eb6b49da37de7ff70a42a09eeb2d8a101792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326) 2024-03-26T21:58:33.971 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:33 smithi138 podman[78594]: 2024-03-26 21:58:33.789633517 +0000 UTC m=+0.820888637 container cleanup 985d4c5f9dcb5251e5704a968ce3eb6b49da37de7ff70a42a09eeb2d8a101792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T21:58:33.972 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:33 smithi138 bash[78594]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b 2024-03-26T21:58:34.213 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T21:58:34.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:33 smithi138 podman[78613]: 2024-03-26 21:58:33.975015473 +0000 UTC m=+0.687611856 container remove 985d4c5f9dcb5251e5704a968ce3eb6b49da37de7ff70a42a09eeb2d8a101792 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mon-b, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:58:34.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:34 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-26T21:58:34.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:34 smithi138 systemd[1]: Stopped Ceph mon.b for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:58:34.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 26 21:58:34 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mon.b.service: Consumed 34.711s CPU time. 2024-03-26T21:58:34.264 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:58:34.265 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T21:58:34.265 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T21:58:34.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.y 2024-03-26T21:58:34.553 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:34 smithi088 systemd[1]: Stopping Ceph mgr.y for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:34.851 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:34 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:58:34.555+0000 7fab089a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:58:34.852 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:34 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y[30334]: 2024-03-26T21:58:34.555+0000 7fab089a0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T21:58:34.852 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:34 smithi088 podman[157407]: 2024-03-26 21:58:34.587485852 +0000 UTC m=+0.124190765 container died c1848489286877ee9f2a68c12da1cd5071d063257c79dc9bbf5a6f186fa526ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-26T21:58:35.202 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:34 smithi088 podman[157407]: 2024-03-26 21:58:34.979205252 +0000 UTC m=+0.515910152 container cleanup c1848489286877ee9f2a68c12da1cd5071d063257c79dc9bbf5a6f186fa526ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T21:58:35.202 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Mar 26 21:58:34 smithi088 bash[157407]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-y 2024-03-26T21:58:35.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T21:58:35.339 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:58:35.339 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T21:58:35.339 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T21:58:35.339 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.x 2024-03-26T21:58:35.684 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:58:35 smithi138 systemd[1]: Stopping Ceph mgr.x for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:35.978 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:58:35 smithi138 podman[78727]: 2024-03-26 21:58:35.686931143 +0000 UTC m=+0.147394753 container died 216f1d336fb9eb530df0132417c2156d2ad42cb4bb40bcef35c65080022c15a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T21:58:36.310 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:58:36 smithi138 podman[78727]: 2024-03-26 21:58:36.120482299 +0000 UTC m=+0.580945907 container cleanup 216f1d336fb9eb530df0132417c2156d2ad42cb4bb40bcef35c65080022c15a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-26T21:58:36.310 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 26 21:58:36 smithi138 bash[78727]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-mgr-x 2024-03-26T21:58:36.534 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T21:58:36.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:58:36.571 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T21:58:36.571 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T21:58:36.572 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.0 2024-03-26T21:58:36.866 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:36 smithi088 systemd[1]: Stopping Ceph osd.0 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:37.237 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:36 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0[43605]: 2024-03-26T21:58:36.868+0000 7f0ef2c4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:58:37.237 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:36 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0[43605]: 2024-03-26T21:58:36.868+0000 7f0ef2c4b700 -1 osd.0 830 *** Got signal Terminated *** 2024-03-26T21:58:37.237 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:36 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0[43605]: 2024-03-26T21:58:36.868+0000 7f0ef2c4b700 -1 osd.0 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:58:42.488 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:42 smithi088 podman[157540]: 2024-03-26 21:58:42.176853653 +0000 UTC m=+5.407128599 container died 1fda4e6cd9a6605f86ee3efb81f6b8957305575eeab8b1239233e8e56bb5a10e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T21:58:42.788 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:42 smithi088 podman[157540]: 2024-03-26 21:58:42.615317589 +0000 UTC m=+5.845592531 container cleanup 1fda4e6cd9a6605f86ee3efb81f6b8957305575eeab8b1239233e8e56bb5a10e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T21:58:42.788 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:42 smithi088 bash[157540]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0 2024-03-26T21:58:43.186 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:42 smithi088 podman[157563]: 2024-03-26 21:58:42.791406605 +0000 UTC m=+0.608820058 container remove 1fda4e6cd9a6605f86ee3efb81f6b8957305575eeab8b1239233e8e56bb5a10e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True) 2024-03-26T21:58:43.487 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:43 smithi088 podman[157659]: 2024-03-26 21:58:43.092024112 +0000 UTC m=+0.017871343 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:58:43.488 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:43 smithi088 podman[157659]: 2024-03-26 21:58:43.192257461 +0000 UTC m=+0.118104677 container create fbca9185f7604cffa58df978d4c673d61197f202a2748c70f4939dd75f91df6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T21:58:43.488 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:43 smithi088 podman[157659]: 2024-03-26 21:58:43.429128071 +0000 UTC m=+0.354975289 container init fbca9185f7604cffa58df978d4c673d61197f202a2748c70f4939dd75f91df6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T21:58:43.488 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:43 smithi088 podman[157659]: 2024-03-26 21:58:43.435158997 +0000 UTC m=+0.361006214 container start fbca9185f7604cffa58df978d4c673d61197f202a2748c70f4939dd75f91df6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T21:58:43.987 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:43 smithi088 podman[157659]: 2024-03-26 21:58:43.520052123 +0000 UTC m=+0.445899349 container attach fbca9185f7604cffa58df978d4c673d61197f202a2748c70f4939dd75f91df6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:58:43.988 INFO:journalctl@ceph.osd.0.smithi088.stdout:Mar 26 21:58:43 smithi088 podman[157659]: 2024-03-26 21:58:43.723470901 +0000 UTC m=+0.649318119 container died fbca9185f7604cffa58df978d4c673d61197f202a2748c70f4939dd75f91df6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0) 2024-03-26T21:58:44.394 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T21:58:44.435 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:58:44.435 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T21:58:44.435 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T21:58:44.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.1 2024-03-26T21:58:44.655 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:44 smithi088 systemd[1]: Stopping Ceph osd.1 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:44.987 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1[49547]: 2024-03-26T21:58:44.771+0000 7f42c7755700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:58:44.988 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1[49547]: 2024-03-26T21:58:44.771+0000 7f42c7755700 -1 osd.1 830 *** Got signal Terminated *** 2024-03-26T21:58:44.988 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:44 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1[49547]: 2024-03-26T21:58:44.771+0000 7f42c7755700 -1 osd.1 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:58:50.478 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:58:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:58:50.085Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:58:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:58:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:58:50.088Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:58:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:58:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:58:50.088Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:58:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:58:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:58:50.088Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:58:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:58:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:58:50.089Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:58:50.487 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:50 smithi088 podman[157790]: 2024-03-26 21:58:50.187660912 +0000 UTC m=+5.529884123 container died 4d1c234903f9849f0ef51b9f49563159fbb94eef074225792d418c894bba02a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True) 2024-03-26T21:58:50.841 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:50 smithi088 podman[157790]: 2024-03-26 21:58:50.592702618 +0000 UTC m=+5.934925873 container cleanup 4d1c234903f9849f0ef51b9f49563159fbb94eef074225792d418c894bba02a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T21:58:50.841 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:50 smithi088 bash[157790]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1 2024-03-26T21:58:51.147 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:50 smithi088 podman[157828]: 2024-03-26 21:58:50.844168839 +0000 UTC m=+0.656364178 container remove 4d1c234903f9849f0ef51b9f49563159fbb94eef074225792d418c894bba02a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:58:51.419 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:51 smithi088 podman[157926]: 2024-03-26 21:58:51.05315475 +0000 UTC m=+0.017796624 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:58:51.419 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:51 smithi088 podman[157926]: 2024-03-26 21:58:51.169992145 +0000 UTC m=+0.134633992 container create 64e2ea323844a347b71183108052bad3212a4478e572236c6d2405ee7a280a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T21:58:51.719 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:51 smithi088 podman[157926]: 2024-03-26 21:58:51.422404325 +0000 UTC m=+0.387046179 container init 64e2ea323844a347b71183108052bad3212a4478e572236c6d2405ee7a280a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-26T21:58:51.719 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:51 smithi088 podman[157926]: 2024-03-26 21:58:51.428705562 +0000 UTC m=+0.393347419 container start 64e2ea323844a347b71183108052bad3212a4478e572236c6d2405ee7a280a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T21:58:51.719 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:51 smithi088 podman[157926]: 2024-03-26 21:58:51.530418238 +0000 UTC m=+0.495060090 container attach 64e2ea323844a347b71183108052bad3212a4478e572236c6d2405ee7a280a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:58:51.987 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:51 smithi088 podman[157926]: 2024-03-26 21:58:51.723135259 +0000 UTC m=+0.687777110 container died 64e2ea323844a347b71183108052bad3212a4478e572236c6d2405ee7a280a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-03-26T21:58:52.268 INFO:journalctl@ceph.osd.1.smithi088.stdout:Mar 26 21:58:52 smithi088 podman[157971]: 2024-03-26 21:58:52.250668933 +0000 UTC m=+0.522008540 container remove 64e2ea323844a347b71183108052bad3212a4478e572236c6d2405ee7a280a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T21:58:52.274 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T21:58:52.315 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:58:52.315 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T21:58:52.315 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T21:58:52.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.2 2024-03-26T21:58:52.612 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:52 smithi088 systemd[1]: Stopping Ceph osd.2 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:58:52.987 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2[56021]: 2024-03-26T21:58:52.614+0000 7f0c36cf8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:58:52.988 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2[56021]: 2024-03-26T21:58:52.614+0000 7f0c36cf8700 -1 osd.2 830 *** Got signal Terminated *** 2024-03-26T21:58:52.988 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:52 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2[56021]: 2024-03-26T21:58:52.614+0000 7f0c36cf8700 -1 osd.2 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:58:58.332 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:58 smithi088 podman[158056]: 2024-03-26 21:58:58.035722384 +0000 UTC m=+5.501133592 container died c4eb15dcd03f4ecb0ebf7d9bec9160d547493b7ab31e15edb88e168b3260039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS) 2024-03-26T21:58:58.583 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:58 smithi088 podman[158056]: 2024-03-26 21:58:58.335800007 +0000 UTC m=+5.801211219 container cleanup c4eb15dcd03f4ecb0ebf7d9bec9160d547493b7ab31e15edb88e168b3260039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:58:58.584 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:58 smithi088 bash[158056]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2 2024-03-26T21:58:58.902 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:58 smithi088 podman[158080]: 2024-03-26 21:58:58.586928623 +0000 UTC m=+0.547062704 container remove c4eb15dcd03f4ecb0ebf7d9bec9160d547493b7ab31e15edb88e168b3260039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5) 2024-03-26T21:58:59.220 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:58 smithi088 podman[158173]: 2024-03-26 21:58:58.808898055 +0000 UTC m=+0.017933622 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:58:59.220 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:58 smithi088 podman[158173]: 2024-03-26 21:58:58.921043508 +0000 UTC m=+0.130079062 container create 253f50b7cd2ab7f25a4305916cb1da5162e755da7d460be1cf3b4171ad1f0036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:58:59.220 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:59 smithi088 podman[158173]: 2024-03-26 21:58:59.148233327 +0000 UTC m=+0.357268881 container init 253f50b7cd2ab7f25a4305916cb1da5162e755da7d460be1cf3b4171ad1f0036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:58:59.220 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:59 smithi088 podman[158173]: 2024-03-26 21:58:59.153446134 +0000 UTC m=+0.362481688 container start 253f50b7cd2ab7f25a4305916cb1da5162e755da7d460be1cf3b4171ad1f0036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T21:58:59.220 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:59 smithi088 podman[158173]: 2024-03-26 21:58:59.223071964 +0000 UTC m=+0.432107522 container attach 253f50b7cd2ab7f25a4305916cb1da5162e755da7d460be1cf3b4171ad1f0036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5) 2024-03-26T21:58:59.487 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 26 21:58:59 smithi088 podman[158173]: 2024-03-26 21:58:59.444843948 +0000 UTC m=+0.653879493 container died 253f50b7cd2ab7f25a4305916cb1da5162e755da7d460be1cf3b4171ad1f0036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T21:59:00.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T21:59:00.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:00.106 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T21:59:00.106 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T21:59:00.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.3 2024-03-26T21:59:00.438 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:00 smithi088 systemd[1]: Stopping Ceph osd.3 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:00.737 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:00 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3[62192]: 2024-03-26T21:59:00.439+0000 7f65a1116700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:59:00.737 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:00 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3[62192]: 2024-03-26T21:59:00.439+0000 7f65a1116700 -1 osd.3 830 *** Got signal Terminated *** 2024-03-26T21:59:00.737 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:00 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3[62192]: 2024-03-26T21:59:00.439+0000 7f65a1116700 -1 osd.3 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:59:03.660 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:03 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:03.350+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:03.928 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:03 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:03.662+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:04.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:04.081+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:04.647 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:04.374+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:04.978 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:04 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:04.649+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:05.237 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:04 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3[62192]: 2024-03-26T21:59:04.863+0000 7f6599702700 -1 osd.3 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.260872+0000 front 2024-03-26T21:58:39.260812+0000 (oldest deadline 2024-03-26T21:59:03.960891+0000) 2024-03-26T21:59:05.415 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:05 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:05.039+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:05.728 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:05 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:05.417+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:05.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:05 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:05.618+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:06.051 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:05 smithi088 podman[158302]: 2024-03-26 21:59:05.802583778 +0000 UTC m=+5.468106754 container died 9ae60174aaeb3eb3f35e9fa61ea3e4550463f9d34240116126f118f36b7f6d89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux ) 2024-03-26T21:59:06.426 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:06 smithi088 podman[158302]: 2024-03-26 21:59:06.152920105 +0000 UTC m=+5.818443082 container cleanup 9ae60174aaeb3eb3f35e9fa61ea3e4550463f9d34240116126f118f36b7f6d89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD) 2024-03-26T21:59:06.426 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:06 smithi088 bash[158302]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3 2024-03-26T21:59:06.457 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:06 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:06.041+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:06.728 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:06 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:06.460+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:06.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:06 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:06.650+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:06.737 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:06 smithi088 podman[158325]: 2024-03-26 21:59:06.429112045 +0000 UTC m=+0.622629573 container remove 9ae60174aaeb3eb3f35e9fa61ea3e4550463f9d34240116126f118f36b7f6d89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T21:59:07.070 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:06 smithi088 podman[158423]: 2024-03-26 21:59:06.673178724 +0000 UTC m=+0.017661663 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:59:07.071 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:06 smithi088 podman[158423]: 2024-03-26 21:59:06.77983496 +0000 UTC m=+0.124317888 container create 9baeafe072ef4f58ab51d5b59d3315720a2adea65524299d54b960ace63254f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:59:07.071 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:06 smithi088 podman[158423]: 2024-03-26 21:59:06.99072347 +0000 UTC m=+0.335206399 container init 9baeafe072ef4f58ab51d5b59d3315720a2adea65524299d54b960ace63254f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T21:59:07.071 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:06 smithi088 podman[158423]: 2024-03-26 21:59:06.996099679 +0000 UTC m=+0.340582615 container start 9baeafe072ef4f58ab51d5b59d3315720a2adea65524299d54b960ace63254f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-26T21:59:07.071 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:07 smithi088 podman[158423]: 2024-03-26 21:59:07.073915338 +0000 UTC m=+0.418398289 container attach 9baeafe072ef4f58ab51d5b59d3315720a2adea65524299d54b960ace63254f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:59:07.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:07 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:07.015+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:07.487 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:07 smithi088 podman[158423]: 2024-03-26 21:59:07.280413258 +0000 UTC m=+0.624896190 container died 9baeafe072ef4f58ab51d5b59d3315720a2adea65524299d54b960ace63254f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T21:59:07.860 INFO:journalctl@ceph.osd.3.smithi088.stdout:Mar 26 21:59:07 smithi088 podman[158470]: 2024-03-26 21:59:07.843734924 +0000 UTC m=+0.558590175 container remove 9baeafe072ef4f58ab51d5b59d3315720a2adea65524299d54b960ace63254f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T21:59:07.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T21:59:07.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:07.907 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T21:59:07.907 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T21:59:07.908 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.4 2024-03-26T21:59:07.939 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:07 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:07.495+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:07.941 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:07 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:07.530+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.745335+0000 front 2024-03-26T21:58:39.745135+0000 (oldest deadline 2024-03-26T21:59:07.345878+0000) 2024-03-26T21:59:07.942 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:07 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:07.682+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:08.228 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:07 smithi138 systemd[1]: Stopping Ceph osd.4 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:08.229 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:07 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:07.989+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:08.527 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:08.234+0000 7fbe3f9f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:59:08.527 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:08.234+0000 7fbe3f9f5700 -1 osd.4 830 *** Got signal Terminated *** 2024-03-26T21:59:08.527 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:08.234+0000 7fbe3f9f5700 -1 osd.4 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:59:08.951 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:08.531+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.745335+0000 front 2024-03-26T21:58:39.745135+0000 (oldest deadline 2024-03-26T21:59:07.345878+0000) 2024-03-26T21:59:08.951 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:08.529+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:08.952 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:08.645+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:09.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:08 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:08.953+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:09.978 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:09 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:09.560+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:09.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:09 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:09.666+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:09.979 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:09 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:09.525+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.745335+0000 front 2024-03-26T21:58:39.745135+0000 (oldest deadline 2024-03-26T21:59:07.345878+0000) 2024-03-26T21:59:10.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:09 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:09.991+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:10.978 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:10.595+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:10.978 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:10.621+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:10.979 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:10.481+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.745335+0000 front 2024-03-26T21:58:39.745135+0000 (oldest deadline 2024-03-26T21:59:07.345878+0000) 2024-03-26T21:59:10.979 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:10 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:10.481+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.346497+0000 front 2024-03-26T21:58:47.346574+0000 (oldest deadline 2024-03-26T21:59:10.246483+0000) 2024-03-26T21:59:11.439 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:11.013+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:11.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:11.622+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:11.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:11.609+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:11.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:11.609+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:11.729 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:11.441+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.745335+0000 front 2024-03-26T21:58:39.745135+0000 (oldest deadline 2024-03-26T21:59:07.345878+0000) 2024-03-26T21:59:11.729 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:11.441+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.346497+0000 front 2024-03-26T21:58:47.346574+0000 (oldest deadline 2024-03-26T21:59:10.246483+0000) 2024-03-26T21:59:12.468 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:11.981+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:12.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:12.585+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:12.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:12.585+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:12.729 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:12.470+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.745335+0000 front 2024-03-26T21:58:39.745135+0000 (oldest deadline 2024-03-26T21:59:07.345878+0000) 2024-03-26T21:59:12.729 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4[39130]: 2024-03-26T21:59:12.470+0000 7fbe3bfe9700 -1 osd.4 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.346497+0000 front 2024-03-26T21:58:47.346574+0000 (oldest deadline 2024-03-26T21:59:10.246483+0000) 2024-03-26T21:59:12.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:12.583+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:12.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:12.583+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:13.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:12.978+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:13.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:12 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:12.978+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:13.978 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:13 smithi138 podman[78877]: 2024-03-26 21:59:13.767625986 +0000 UTC m=+5.661743227 container died 67a71a6acb7fa9e0c1c4c921824b03dd7609fc24121f6030a7bc81e48d2d303a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T21:59:13.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:13 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:13.615+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:13.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:13 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:13.615+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:13.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:13 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:13.567+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:13.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:13 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:13.567+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:14.363 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:13 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:13.995+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:14.364 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:13 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:13.995+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:14.364 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:14 smithi138 podman[78877]: 2024-03-26 21:59:14.174189558 +0000 UTC m=+6.068306800 container cleanup 67a71a6acb7fa9e0c1c4c921824b03dd7609fc24121f6030a7bc81e48d2d303a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:59:14.364 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:14 smithi138 bash[78877]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4 2024-03-26T21:59:14.728 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:14 smithi138 podman[78899]: 2024-03-26 21:59:14.400852718 +0000 UTC m=+0.625230605 container remove 67a71a6acb7fa9e0c1c4c921824b03dd7609fc24121f6030a7bc81e48d2d303a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T21:59:14.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:14 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:14.547+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:14.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:14 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:14.547+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:14.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:14 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:14.568+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:14.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:14 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:14.568+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:15.001 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:14 smithi138 podman[78993]: 2024-03-26 21:59:14.636906432 +0000 UTC m=+0.018023992 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:59:15.001 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:14 smithi138 podman[78993]: 2024-03-26 21:59:14.76045533 +0000 UTC m=+0.141572877 container create e0b0beb986c7bf5561327240f135a82a02038a467fbf048006c44bedd7e35851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-26T21:59:15.298 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:15 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:15.003+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:15.298 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:15 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:15.003+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:15.298 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:15 smithi138 podman[78993]: 2024-03-26 21:59:15.005498226 +0000 UTC m=+0.386615774 container init e0b0beb986c7bf5561327240f135a82a02038a467fbf048006c44bedd7e35851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T21:59:15.298 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:15 smithi138 podman[78993]: 2024-03-26 21:59:15.011162418 +0000 UTC m=+0.392279974 container start e0b0beb986c7bf5561327240f135a82a02038a467fbf048006c44bedd7e35851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-26T21:59:15.298 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:15 smithi138 podman[78993]: 2024-03-26 21:59:15.097046791 +0000 UTC m=+0.478164368 container attach e0b0beb986c7bf5561327240f135a82a02038a467fbf048006c44bedd7e35851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2) 2024-03-26T21:59:15.562 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 26 21:59:15 smithi138 podman[78993]: 2024-03-26 21:59:15.301239169 +0000 UTC m=+0.682356717 container died e0b0beb986c7bf5561327240f135a82a02038a467fbf048006c44bedd7e35851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-03-26T21:59:15.562 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:15 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:15.530+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:15.562 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:15 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:15.530+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:15.832 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:15 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:15.564+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:15.832 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:15 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:15.564+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:15.859 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T21:59:15.940 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:15.940 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T21:59:15.940 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T21:59:15.940 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.5 2024-03-26T21:59:16.228 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:16 smithi138 systemd[1]: Stopping Ceph osd.5 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:16.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:16.027+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:16.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:16.027+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:16.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:16.523+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:16.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:16.523+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:16.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:16.282+0000 7fe0b6342700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:59:16.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:16.282+0000 7fe0b6342700 -1 osd.5 830 *** Got signal Terminated *** 2024-03-26T21:59:16.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:16.282+0000 7fe0b6342700 -1 osd.5 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:59:16.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:16.519+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:16.729 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:16 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:16.519+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:17.479 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:17.069+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:17.479 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:17.069+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:17.978 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:17.546+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:17.978 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:17.546+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:17.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:17.502+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:17.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:17 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:17.502+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:18.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:18.025+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:18.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:18.025+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:18.978 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:18.526+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:18.978 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:18.526+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:18.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:18.501+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:18.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:18.501+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:18.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:18 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:18.501+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:19.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:19.015+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:19.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:19.015+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:19.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:19.015+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:19.978 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:19.509+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:19.978 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:19.509+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:19.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:19.509+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:19.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:19.565+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:19.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:19.565+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:19.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:19 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:19.565+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:55.869996+0000 front 2024-03-26T21:58:55.869982+0000 (oldest deadline 2024-03-26T21:59:18.769864+0000) 2024-03-26T21:59:20.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:20.056+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:20.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:20.056+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:20.479 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:20.056+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:20.978 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:20.482+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:20.978 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:20.482+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:20.979 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:20.482+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:20.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:20.583+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:37.266557+0000 front 2024-03-26T21:58:37.266572+0000 (oldest deadline 2024-03-26T21:59:03.166391+0000) 2024-03-26T21:59:20.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:20.583+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:47.168363+0000 front 2024-03-26T21:58:47.168471+0000 (oldest deadline 2024-03-26T21:59:10.668333+0000) 2024-03-26T21:59:20.979 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:20 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5[45139]: 2024-03-26T21:59:20.583+0000 7fe0ae92e700 -1 osd.5 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:55.869996+0000 front 2024-03-26T21:58:55.869982+0000 (oldest deadline 2024-03-26T21:59:18.769864+0000) 2024-03-26T21:59:21.441 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:21.035+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:21.441 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:21.035+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:21.441 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:21.035+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:21.718 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:21.443+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:21.718 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:21.443+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:21.718 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:21 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:21.443+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:21.978 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:21 smithi138 podman[79125]: 2024-03-26 21:59:21.720776945 +0000 UTC m=+5.552599511 container died e5299b4c59d5c405e39618836687b0599b53421f2d2032853741f51373b49428 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T21:59:22.387 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:22 smithi138 podman[79125]: 2024-03-26 21:59:22.097043896 +0000 UTC m=+5.928866466 container cleanup e5299b4c59d5c405e39618836687b0599b53421f2d2032853741f51373b49428 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:59:22.387 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:22 smithi138 bash[79125]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5 2024-03-26T21:59:22.387 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:22.009+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:22.387 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:22.009+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:22.387 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:22.009+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:22.728 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:22 smithi138 podman[79147]: 2024-03-26 21:59:22.390132259 +0000 UTC m=+0.668382529 container remove e5299b4c59d5c405e39618836687b0599b53421f2d2032853741f51373b49428 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:59:22.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:22.402+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:22.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:22.402+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:22.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:22.402+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:23.066 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:22.980+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:23.066 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:22.980+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:23.066 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:22 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:22.980+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:23.066 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:22 smithi138 podman[79241]: 2024-03-26 21:59:22.65104852 +0000 UTC m=+0.018152539 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:59:23.067 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:22 smithi138 podman[79241]: 2024-03-26 21:59:22.766231461 +0000 UTC m=+0.133335451 container create d52c30aca04a6c695f33138bfa4c89d402939d85bb4b2c3416cb6f37c2178683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T21:59:23.067 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:22 smithi138 podman[79241]: 2024-03-26 21:59:22.97810637 +0000 UTC m=+0.345210375 container init d52c30aca04a6c695f33138bfa4c89d402939d85bb4b2c3416cb6f37c2178683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T21:59:23.067 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:22 smithi138 podman[79241]: 2024-03-26 21:59:22.9841649 +0000 UTC m=+0.351268894 container start d52c30aca04a6c695f33138bfa4c89d402939d85bb4b2c3416cb6f37c2178683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T21:59:23.067 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:23 smithi138 podman[79241]: 2024-03-26 21:59:23.069309146 +0000 UTC m=+0.436413158 container attach d52c30aca04a6c695f33138bfa4c89d402939d85bb4b2c3416cb6f37c2178683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:59:23.353 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 26 21:59:23 smithi138 podman[79241]: 2024-03-26 21:59:23.275038288 +0000 UTC m=+0.642142299 container died d52c30aca04a6c695f33138bfa4c89d402939d85bb4b2c3416cb6f37c2178683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-26T21:59:23.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:23 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:23.355+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:23.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:23 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:23.355+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:23.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:23 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:23.355+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:23.868 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T21:59:23.911 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:23.911 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T21:59:23.911 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T21:59:23.911 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.6 2024-03-26T21:59:24.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:23 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:23.950+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:24.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:23 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:23.950+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:24.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:23 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:23.950+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:24.229 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:24 smithi138 systemd[1]: Stopping Ceph osd.6 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:24.729 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:24.256+0000 7f4267276700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:59:24.729 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:24.256+0000 7f4267276700 -1 osd.6 830 *** Got signal Terminated *** 2024-03-26T21:59:24.729 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:24.256+0000 7f4267276700 -1 osd.6 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:59:24.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:24.334+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:24.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:24.334+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:24.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:24.334+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:25.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:24.928+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:25.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:24.928+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:25.229 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:24 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:24.928+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:25.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:25 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:25.325+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:25.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:25 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:25.325+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:25.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:25 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:25.325+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:26.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:25 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:25.940+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:26.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:25 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:25.940+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:26.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:25 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:25.940+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:26.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:26 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:26.324+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:26.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:26 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:26.324+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:26.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:26 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:26.324+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:27.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:26 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:26.902+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:27.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:26 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:26.902+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:27.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:26 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:26.902+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:27.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:27 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:27.306+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:27.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:27 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:27.306+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:27.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:27 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:27.306+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:27.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:27 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:27.306+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:28.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:27 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:27.898+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:28.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:27 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:27.898+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:28.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:27 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:27.898+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:28.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:28 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:28.299+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:28.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:28 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:28.299+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:28.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:28 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:28.299+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:28.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:28 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:28.299+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:29.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:28 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:28.875+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.575833+0000 front 2024-03-26T21:58:39.575807+0000 (oldest deadline 2024-03-26T21:59:03.075636+0000) 2024-03-26T21:59:29.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:28 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:28.875+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:46.878118+0000 front 2024-03-26T21:58:46.878045+0000 (oldest deadline 2024-03-26T21:59:12.177992+0000) 2024-03-26T21:59:29.228 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:28 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6[51082]: 2024-03-26T21:59:28.875+0000 7f426386a700 -1 osd.6 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:57.279917+0000 front 2024-03-26T21:58:57.279943+0000 (oldest deadline 2024-03-26T21:59:18.979855+0000) 2024-03-26T21:59:29.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:29 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:29.277+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:29.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:29 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:29.277+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:29.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:29 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:29.277+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:29.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:29 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:29.277+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:30.050 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:29 smithi138 podman[79371]: 2024-03-26 21:59:29.784146218 +0000 UTC m=+5.642027723 container died f3bc2842c2bfe9c400ba0f17dc030bddae8fb70c183850de86d6da29f98b00f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T21:59:30.459 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:30 smithi138 podman[79371]: 2024-03-26 21:59:30.151792149 +0000 UTC m=+6.009673659 container cleanup f3bc2842c2bfe9c400ba0f17dc030bddae8fb70c183850de86d6da29f98b00f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T21:59:30.459 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:30 smithi138 bash[79371]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6 2024-03-26T21:59:30.459 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:30.250+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:30.459 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:30.250+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:30.459 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:30.250+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:30.459 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:30 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:30.250+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:30.728 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:30 smithi138 podman[79393]: 2024-03-26 21:59:30.461980204 +0000 UTC m=+0.674187382 container remove f3bc2842c2bfe9c400ba0f17dc030bddae8fb70c183850de86d6da29f98b00f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:59:31.152 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:30 smithi138 podman[79494]: 2024-03-26 21:59:30.727014841 +0000 UTC m=+0.018027994 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:59:31.153 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:30 smithi138 podman[79494]: 2024-03-26 21:59:30.837998945 +0000 UTC m=+0.129012071 container create 638cf8ab4a8ee258c5efd2e0fb63c63224affb8f4e471dc15442016e363a1026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T21:59:31.153 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:31 smithi138 podman[79494]: 2024-03-26 21:59:31.04083495 +0000 UTC m=+0.331848086 container init 638cf8ab4a8ee258c5efd2e0fb63c63224affb8f4e471dc15442016e363a1026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-26T21:59:31.153 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:31 smithi138 podman[79494]: 2024-03-26 21:59:31.047210935 +0000 UTC m=+0.338224064 container start 638cf8ab4a8ee258c5efd2e0fb63c63224affb8f4e471dc15442016e363a1026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T21:59:31.153 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:31 smithi138 podman[79494]: 2024-03-26 21:59:31.155424777 +0000 UTC m=+0.446437921 container attach 638cf8ab4a8ee258c5efd2e0fb63c63224affb8f4e471dc15442016e363a1026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-26T21:59:31.478 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 26 21:59:31 smithi138 podman[79494]: 2024-03-26 21:59:31.339635554 +0000 UTC m=+0.630648689 container died 638cf8ab4a8ee258c5efd2e0fb63c63224affb8f4e471dc15442016e363a1026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:59:31.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:31.224+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:31.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:31.224+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:31.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:31.224+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:31.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:31 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:31.224+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:31.926 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T21:59:31.966 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:31.967 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T21:59:31.967 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T21:59:31.967 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.7 2024-03-26T21:59:32.190 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 systemd[1]: Stopping Ceph osd.7 for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:32.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:32.191+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:32.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:32.192+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:32.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:32.192+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:32.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:32.192+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:32.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:32.317+0000 7f442ee8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:59:32.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:32.317+0000 7f442ee8d700 -1 osd.7 830 *** Got signal Terminated *** 2024-03-26T21:59:32.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:32 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:32.317+0000 7f442ee8d700 -1 osd.7 830 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:59:33.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:33 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:33.220+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:33.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:33 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:33.220+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:33.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:33 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:33.220+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:33.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:33 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:33.220+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:34.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:34 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:34.196+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:34.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:34 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:34.196+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:34.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:34 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:34.196+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:34.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:34 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:34.196+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:35.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:35 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:35.210+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:35.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:35 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:35.210+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:35.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:35 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:35.210+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:35.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:35 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:35.210+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:35.479 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:35 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:35.210+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.138:6802 osd.4 since back 2024-03-26T21:59:11.709904+0000 front 2024-03-26T21:59:11.709766+0000 (oldest deadline 2024-03-26T21:59:34.609754+0000) 2024-03-26T21:59:36.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:36 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:36.259+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:36.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:36 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:36.259+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:36.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:36 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:36.259+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:36.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:36 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:36.259+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:36.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:36 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:36.259+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.138:6802 osd.4 since back 2024-03-26T21:59:11.709904+0000 front 2024-03-26T21:59:11.709766+0000 (oldest deadline 2024-03-26T21:59:34.609754+0000) 2024-03-26T21:59:37.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:37.248+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6803 osd.0 since back 2024-03-26T21:58:39.804684+0000 front 2024-03-26T21:58:39.804545+0000 (oldest deadline 2024-03-26T21:59:02.704589+0000) 2024-03-26T21:59:37.728 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:37.248+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6807 osd.1 since back 2024-03-26T21:58:48.406857+0000 front 2024-03-26T21:58:48.406681+0000 (oldest deadline 2024-03-26T21:59:12.506702+0000) 2024-03-26T21:59:37.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:37.248+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6811 osd.2 since back 2024-03-26T21:58:52.507271+0000 front 2024-03-26T21:58:52.507214+0000 (oldest deadline 2024-03-26T21:59:18.407084+0000) 2024-03-26T21:59:37.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:37.248+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.88:6815 osd.3 since back 2024-03-26T21:59:01.208566+0000 front 2024-03-26T21:59:01.208529+0000 (oldest deadline 2024-03-26T21:59:27.108416+0000) 2024-03-26T21:59:37.729 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:37 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7[57526]: 2024-03-26T21:59:37.248+0000 7f442b481700 -1 osd.7 830 heartbeat_check: no reply from 172.21.15.138:6802 osd.4 since back 2024-03-26T21:59:11.709904+0000 front 2024-03-26T21:59:11.709766+0000 (oldest deadline 2024-03-26T21:59:34.609754+0000) 2024-03-26T21:59:38.096 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:37 smithi138 podman[79626]: 2024-03-26 21:59:37.846981303 +0000 UTC m=+5.660086800 container died aa38531508eaef181e1273365975682d44dfdae3e3416f6377bb2d429dac552c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326) 2024-03-26T21:59:38.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:38 smithi138 podman[79626]: 2024-03-26 21:59:38.198633973 +0000 UTC m=+6.011739467 container cleanup aa38531508eaef181e1273365975682d44dfdae3e3416f6377bb2d429dac552c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T21:59:38.478 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:38 smithi138 bash[79626]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7 2024-03-26T21:59:38.818 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:38 smithi138 podman[79660]: 2024-03-26 21:59:38.482800455 +0000 UTC m=+0.637201381 container remove aa38531508eaef181e1273365975682d44dfdae3e3416f6377bb2d429dac552c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T21:59:39.083 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:38 smithi138 podman[79759]: 2024-03-26 21:59:38.724070202 +0000 UTC m=+0.017957857 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:59:39.084 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:38 smithi138 podman[79759]: 2024-03-26 21:59:38.825254256 +0000 UTC m=+0.119141888 container create 52e500c6ee41bf18bc9fe5d4dfe1498f3b03cbd9cb218ee45844b9b12f8b7bca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:59:39.084 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:39 smithi138 podman[79759]: 2024-03-26 21:59:39.086513287 +0000 UTC m=+0.380400921 container init 52e500c6ee41bf18bc9fe5d4dfe1498f3b03cbd9cb218ee45844b9b12f8b7bca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-26T21:59:39.382 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:39 smithi138 podman[79759]: 2024-03-26 21:59:39.092474099 +0000 UTC m=+0.386361731 container start 52e500c6ee41bf18bc9fe5d4dfe1498f3b03cbd9cb218ee45844b9b12f8b7bca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T21:59:39.382 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:39 smithi138 podman[79759]: 2024-03-26 21:59:39.177899307 +0000 UTC m=+0.471786970 container attach 52e500c6ee41bf18bc9fe5d4dfe1498f3b03cbd9cb218ee45844b9b12f8b7bca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T21:59:39.703 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 26 21:59:39 smithi138 podman[79759]: 2024-03-26 21:59:39.384932422 +0000 UTC m=+0.678820065 container died 52e500c6ee41bf18bc9fe5d4dfe1498f3b03cbd9cb218ee45844b9b12f8b7bca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T21:59:39.990 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T21:59:40.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:40.031 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T21:59:40.031 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T21:59:40.031 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T21:59:40.315 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:40 smithi088 systemd[1]: Stopping Ceph rgw.foo.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:40.568 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:40 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a[67300]: 2024-03-26T21:59:40.317+0000 7f2ab40ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:59:40.568 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:40 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a[67300]: 2024-03-26T21:59:40.317+0000 7f2ac2972ac0 -1 shutting down 2024-03-26T21:59:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:50.085Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:59:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:50.087Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:59:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:50.088Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:59:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:50.088Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:59:50.479 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:50 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:50.090Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.88:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.88:8765: connect: connection refused" 2024-03-26T21:59:50.637 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:50 smithi088 bash[158554]: time="2024-03-26T21:59:50Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T21:59:50.638 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:50 smithi088 podman[158554]: 2024-03-26 21:59:50.345437785 +0000 UTC m=+10.120560810 container died 2d95ae11981cabc1abf725e0aa6bb26b240e295b3e76458dd98ffe04f3bb52c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326) 2024-03-26T21:59:50.988 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:50 smithi088 podman[158554]: 2024-03-26 21:59:50.805317246 +0000 UTC m=+10.580440272 container cleanup 2d95ae11981cabc1abf725e0aa6bb26b240e295b3e76458dd98ffe04f3bb52c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T21:59:50.988 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:50 smithi088 bash[158554]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T21:59:51.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T21:59:51.249 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:51 smithi088 podman[158591]: 2024-03-26 21:59:51.081249088 +0000 UTC m=+0.731474271 container remove 2d95ae11981cabc1abf725e0aa6bb26b240e295b3e76458dd98ffe04f3bb52c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-rgw-foo-a, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T21:59:51.249 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:51 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T21:59:51.249 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:51 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-26T21:59:51.249 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:51 smithi088 systemd[1]: Stopped Ceph rgw.foo.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:59:51.249 INFO:journalctl@ceph.rgw.foo.a.smithi088.stdout:Mar 26 21:59:51 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@rgw.foo.a.service: Consumed 5.909s CPU time. 2024-03-26T21:59:51.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:51.261 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T21:59:51.261 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T21:59:51.261 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T21:59:51.574 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 systemd[1]: Stopping Ceph prometheus.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.576Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.576Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.576Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.576Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.576Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.576Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.577Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.577Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T21:59:51.864 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.577Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T21:59:51.865 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.577Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T21:59:51.865 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.577Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T21:59:51.865 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a[68124]: ts=2024-03-26T21:59:51.577Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T21:59:51.865 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 podman[79890]: 2024-03-26 21:59:51.587540588 +0000 UTC m=+0.127864313 container died 25814fe936c38eb9e2111cb1cc0182c361a8fa4b0aeebc79d09e96cd6378629a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:52.214 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 podman[79890]: 2024-03-26 21:59:51.974298495 +0000 UTC m=+0.514622290 container cleanup 25814fe936c38eb9e2111cb1cc0182c361a8fa4b0aeebc79d09e96cd6378629a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:52.214 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 26 21:59:51 smithi138 bash[79890]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T21:59:52.389 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T21:59:52.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:59:52.426 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T21:59:52.426 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm rm-cluster --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T21:59:52.617 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:59:54.419 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:54 smithi088 systemd[1]: Stopping Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:54.738 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:54 smithi088 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a[78450]: ts=2024-03-26T21:59:54.420Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:59:54.738 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:54 smithi088 podman[159075]: 2024-03-26 21:59:54.472529573 +0000 UTC m=+0.150354299 container died b92f5c704a604a7435bc1386201add7f64c62446edc46d81cb208a694f045945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:55.099 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:54 smithi088 podman[159075]: 2024-03-26 21:59:54.826325494 +0000 UTC m=+0.504150218 container cleanup b92f5c704a604a7435bc1386201add7f64c62446edc46d81cb208a694f045945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:55.099 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:54 smithi088 bash[159075]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T21:59:55.376 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:55 smithi088 podman[159094]: 2024-03-26 21:59:55.102377561 +0000 UTC m=+0.630050035 container remove b92f5c704a604a7435bc1386201add7f64c62446edc46d81cb208a694f045945 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:55.376 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:55 smithi088 podman[159094]: 2024-03-26 21:59:55.186257643 +0000 UTC m=+0.713930131 volume remove c160c6e406a3ffb795b342d44dc1f57bbd7d0e434ff0eb01688a27ae27e1e627 2024-03-26T21:59:55.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:55 smithi088 systemd[1]: Stopping Ceph node-exporter.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:55.738 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:55 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T21:59:55.738 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:55 smithi088 systemd[1]: Stopped Ceph alertmanager.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:59:55.738 INFO:journalctl@ceph.alertmanager.a.smithi088.stdout:Mar 26 21:59:55 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 2.511s CPU time. 2024-03-26T21:59:56.177 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:55 smithi088 podman[159211]: 2024-03-26 21:59:55.902528388 +0000 UTC m=+0.176680725 container died 594e469269315244e47f081dc7f3f2647c500becc6096f14db197de3a2a4ecbc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:56.177 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:56 smithi088 podman[159211]: 2024-03-26 21:59:56.180490544 +0000 UTC m=+0.454642873 container cleanup 594e469269315244e47f081dc7f3f2647c500becc6096f14db197de3a2a4ecbc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:56.453 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:56 smithi088 bash[159211]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T21:59:56.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:56 smithi088 podman[159230]: 2024-03-26 21:59:56.456426942 +0000 UTC m=+0.558262078 container remove 594e469269315244e47f081dc7f3f2647c500becc6096f14db197de3a2a4ecbc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:59:56.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:56 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T21:59:56.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:56 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T21:59:56.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:56 smithi088 systemd[1]: Stopped Ceph node-exporter.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T21:59:56.738 INFO:journalctl@ceph.node-exporter.a.smithi088.stdout:Mar 26 21:59:56 smithi088 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 5.418s CPU time. 2024-03-26T21:59:57.217 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm rm-cluster --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T21:59:57.410 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T21:59:58.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:59:58 smithi138 systemd[1]: Stopping Ceph iscsi.iscsi.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T21:59:59.228 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 21:59:58 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a[62516]: debug Shutdown received 2024-03-26T22:00:09.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:08 smithi138 bash[80311]: time="2024-03-26T22:00:08Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T22:00:09.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:08 smithi138 podman[80311]: 2024-03-26 22:00:08.915771917 +0000 UTC m=+10.130222784 container died eea1c9d0fd5bd6638eb28146b7321d367e29fccc16714b32ed3129874a9850d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux ) 2024-03-26T22:00:09.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:09 smithi138 podman[80311]: 2024-03-26 22:00:09.266831038 +0000 UTC m=+10.481281905 container cleanup eea1c9d0fd5bd6638eb28146b7321d367e29fccc16714b32ed3129874a9850d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-26T22:00:09.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:09 smithi138 bash[80311]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T22:00:09.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:09 smithi138 podman[80337]: 2024-03-26 22:00:09.476273794 +0000 UTC m=+0.557933469 container remove eea1c9d0fd5bd6638eb28146b7321d367e29fccc16714b32ed3129874a9850d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T22:00:09.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:09 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T22:00:09.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:09 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T22:00:09.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:09 smithi138 systemd[1]: Stopped Ceph iscsi.iscsi.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T22:00:09.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 26 22:00:09 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 4.510s CPU time. 2024-03-26T22:00:11.414 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:11 smithi138 systemd[1]: Stopping Ceph grafana.a for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T22:00:11.729 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=server t=2024-03-26T22:00:11.460641058Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-26T22:00:11.729 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=ticker t=2024-03-26T22:00:11.46103833Z level=info msg=stopped last_tick=2024-03-26T22:00:10Z 2024-03-26T22:00:11.729 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:11 smithi138 ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a[66610]: logger=tracing t=2024-03-26T22:00:11.461081706Z level=info msg="Closing tracing" 2024-03-26T22:00:11.729 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:11 smithi138 podman[80615]: 2024-03-26 22:00:11.471192152 +0000 UTC m=+0.136357722 container died db10ec6efeae2ed2608a3a83e5300e26e4264a7a24f4aaa2ba9d9b6d27f38fad (image=quay.io/ceph/grafana:9.4.12, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T22:00:12.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:11 smithi138 podman[80615]: 2024-03-26 22:00:11.832709449 +0000 UTC m=+0.497875019 container cleanup db10ec6efeae2ed2608a3a83e5300e26e4264a7a24f4aaa2ba9d9b6d27f38fad (image=quay.io/ceph/grafana:9.4.12, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T22:00:12.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:11 smithi138 bash[80615]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a 2024-03-26T22:00:12.395 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:12 smithi138 podman[80634]: 2024-03-26 22:00:12.092028311 +0000 UTC m=+0.618126761 container remove db10ec6efeae2ed2608a3a83e5300e26e4264a7a24f4aaa2ba9d9b6d27f38fad (image=quay.io/ceph/grafana:9.4.12, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T22:00:12.396 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:12 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T22:00:12.396 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:12 smithi138 systemd[1]: Stopped Ceph grafana.a for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T22:00:12.396 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 26 22:00:12 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@grafana.a.service: Consumed 24.572s CPU time. 2024-03-26T22:00:12.705 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:12 smithi138 systemd[1]: Stopping Ceph node-exporter.b for e240a1a4-ebb5-11ee-95d0-87774f69a715... 2024-03-26T22:00:12.960 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:12 smithi138 podman[80763]: 2024-03-26 22:00:12.70849072 +0000 UTC m=+0.118290813 container died 21807d83fb65938fed267db2d486043cc8e202abc6b668f0c685bae55b3db59a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T22:00:13.211 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:13 smithi138 podman[80763]: 2024-03-26 22:00:13.036569633 +0000 UTC m=+0.446369726 container cleanup 21807d83fb65938fed267db2d486043cc8e202abc6b668f0c685bae55b3db59a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T22:00:13.211 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:13 smithi138 bash[80763]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T22:00:13.211 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:13 smithi138 podman[80782]: 2024-03-26 22:00:13.21252232 +0000 UTC m=+0.508826563 container remove 21807d83fb65938fed267db2d486043cc8e202abc6b668f0c685bae55b3db59a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T22:00:13.464 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:13 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T22:00:13.465 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:13 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T22:00:13.465 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:13 smithi138 systemd[1]: Stopped Ceph node-exporter.b for e240a1a4-ebb5-11ee-95d0-87774f69a715. 2024-03-26T22:00:13.465 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 26 22:00:13 smithi138 systemd[1]: ceph-e240a1a4-ebb5-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 5.342s CPU time. 2024-03-26T22:00:14.073 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T22:00:14.109 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T22:00:14.139 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T22:00:14.141 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/remote/smithi088/crash 2024-03-26T22:00:14.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/crash -- . 2024-03-26T22:00:14.193 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T22:00:14.193 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2024-03-26T22:00:14.195 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/remote/smithi138/crash 2024-03-26T22:00:14.196 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/crash -- . 2024-03-26T22:00:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:tar: /var/lib/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T22:00:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:tar: Error is not recoverable: exiting now 2024-03-26T22:00:14.235 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T22:00:14.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T22:00:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:2024-03-26T21:30:00.000178+0000 mon.a (mon.0) 2476 : cluster [WRN] Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T22:00:14.267 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T22:00:14.267 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[SEC\]' /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T22:00:14.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]' /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T22:00:14.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[WRN\]' /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T22:00:14.486 INFO:teuthology.orchestra.run.smithi088.stdout:2024-03-26T21:30:00.000178+0000 mon.a (mon.0) 2476 : cluster [WRN] Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T22:00:14.486 INFO:tasks.cephadm:Compressing logs... 2024-03-26T22:00:14.486 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T22:00:14.528 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T22:00:14.552 INFO:teuthology.orchestra.run.smithi088.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T22:00:14.552 INFO:teuthology.orchestra.run.smithi088.stderr:: No such file or directory 2024-03-26T22:00:14.552 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T22:00:14.552 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T22:00:14.553 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T22:00:14.553 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T22:00:14.553 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T22:00:14.553 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T22:00:14.555 INFO:teuthology.orchestra.run.smithi138.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T22:00:14.555 INFO:teuthology.orchestra.run.smithi138.stderr:: No such file or directory 2024-03-26T22:00:14.556 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T22:00:14.556 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T22:00:14.556 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T22:00:14.556 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T22:00:14.556 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T22:00:14.557 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T22:00:14.559 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.4.log: /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T22:00:14.559 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T22:00:14.559 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T22:00:14.559 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/tcmu-runner.log: 67.3% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T22:00:14.559 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log 2024-03-26T22:00:14.559 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T22:00:14.560 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.3.log: /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mgr.y.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T22:00:14.561 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T22:00:14.567 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log: 88.2% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T22:00:14.567 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T22:00:14.567 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T22:00:14.568 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-volume.log: 92.4% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T22:00:14.570 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T22:00:14.570 INFO:teuthology.orchestra.run.smithi138.stderr: 92.3% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T22:00:14.574 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.0.log: 95.3% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T22:00:14.574 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log 2024-03-26T22:00:14.575 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.cephadm.log: 89.2% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T22:00:14.575 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T22:00:14.582 INFO:teuthology.orchestra.run.smithi138.stderr: 90.8% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T22:00:14.585 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log: 93.5% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T22:00:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T22:00:14.606 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T22:00:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-volume.log: 93.5% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T22:00:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.8% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T22:00:15.109 INFO:teuthology.orchestra.run.smithi088.stderr: 90.1% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T22:00:16.005 INFO:teuthology.orchestra.run.smithi138.stderr: 92.2% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T22:00:16.258 INFO:teuthology.orchestra.run.smithi088.stderr: 92.2% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T22:00:18.274 INFO:teuthology.orchestra.run.smithi088.stderr: 91.6% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T22:00:23.098 INFO:teuthology.orchestra.run.smithi138.stderr: 93.9% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T22:00:23.229 INFO:teuthology.orchestra.run.smithi138.stderr: 93.9% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T22:00:23.278 INFO:teuthology.orchestra.run.smithi088.stderr: 93.9% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T22:00:23.286 INFO:teuthology.orchestra.run.smithi138.stderr: 93.9% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T22:00:23.496 INFO:teuthology.orchestra.run.smithi088.stderr: 94.0% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T22:00:23.729 INFO:teuthology.orchestra.run.smithi088.stderr: 93.9% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T22:00:23.757 INFO:teuthology.orchestra.run.smithi138.stderr: 94.0% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T22:00:23.759 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-26T22:00:23.759 INFO:teuthology.orchestra.run.smithi138.stderr:real 0m9.217s 2024-03-26T22:00:23.759 INFO:teuthology.orchestra.run.smithi138.stderr:user 0m35.357s 2024-03-26T22:00:23.760 INFO:teuthology.orchestra.run.smithi138.stderr:sys 0m1.250s 2024-03-26T22:00:24.386 INFO:teuthology.orchestra.run.smithi088.stderr: 94.0% -- replaced with /var/log/ceph/e240a1a4-ebb5-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T22:00:24.388 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-26T22:00:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m9.848s 2024-03-26T22:00:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m41.140s 2024-03-26T22:00:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m1.438s 2024-03-26T22:00:24.388 INFO:tasks.cephadm:Archiving logs... 2024-03-26T22:00:24.389 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/remote/smithi088/log 2024-03-26T22:00:24.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T22:00:29.952 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/remote/smithi138/log 2024-03-26T22:00:29.953 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T22:00:34.880 INFO:tasks.cephadm:Removing cluster... 2024-03-26T22:00:34.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo cephadm rm-cluster --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 --force 2024-03-26T22:00:35.074 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T22:00:35.373 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm rm-cluster --fsid e240a1a4-ebb5-11ee-95d0-87774f69a715 --force 2024-03-26T22:00:35.568 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: e240a1a4-ebb5-11ee-95d0-87774f69a715 2024-03-26T22:00:35.865 INFO:tasks.cephadm:Teardown complete 2024-03-26T22:00:35.865 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T22:00:35.875 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T22:00:35.875 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T22:00:35.878 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T22:00:35.961 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T22:00:35.969 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T22:00:35.970 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T22:00:35.973 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T22:00:35.984 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-03-26T22:00:35.986 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-03-26T22:00:35.989 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T22:00:35.989 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-03-26T22:00:35.989 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 54 +494us[ +494us] +/- 69ms 2024-03-26T22:00:35.989 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 7 377 128 -945us[ -937us] +/- 79ms 2024-03-26T22:00:35.989 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +686us[ +686us] +/- 69ms 2024-03-26T22:00:35.989 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T22:00:35.990 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T22:00:35.990 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-03-26T22:00:35.990 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +396us[ +397us] +/- 69ms 2024-03-26T22:00:35.990 INFO:teuthology.orchestra.run.smithi138.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -1033us[-1032us] +/- 80ms 2024-03-26T22:00:35.990 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +639us[ +639us] +/- 69ms 2024-03-26T22:00:35.991 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T22:00:35.992 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T22:00:36.002 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T22:00:36.003 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T22:00:36.011 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T22:00:36.112 DEBUG:teuthology.orchestra.run.smithi138:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T22:00:36.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T22:00:36.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:00:36.219 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T22:00:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:00:36.249 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T22:00:36.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T22:00:36.267 INFO:teuthology.task.internal:Duration was 3180.993800 seconds 2024-03-26T22:00:36.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T22:00:36.276 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T22:00:36.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T22:00:36.278 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T22:00:36.319 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T22:00:36.332 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T22:00:36.523 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T22:00:36.523 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-03-26T22:00:36.523 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T22:00:36.583 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-03-26T22:00:36.583 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T22:00:36.613 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T22:00:36.613 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T22:00:36.625 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T22:00:36.681 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T22:00:36.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T22:00:36.694 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T22:00:37.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T22:00:37.138 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T22:00:37.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T22:00:37.165 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T22:00:37.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T22:00:37.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:00:37.208 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:00:37.231 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-03-26T22:00:37.258 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-03-26T22:00:37.294 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:00:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:00:37.308 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:00:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:00:37.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T22:00:37.358 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T22:00:37.358 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/remote/smithi088 2024-03-26T22:00:37.359 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T22:00:37.424 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/remote/smithi138 2024-03-26T22:00:37.425 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T22:00:37.465 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T22:00:37.465 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T22:00:37.467 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T22:00:37.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T22:00:37.525 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T22:00:37.526 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T22:00:37.535 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T22:00:37.535 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T22:00:37.537 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T22:00:37.549 INFO:teuthology.orchestra.run.smithi088.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 22:00 /home/ubuntu/cephtest 2024-03-26T22:00:37.567 INFO:teuthology.orchestra.run.smithi138.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 22:00 /home/ubuntu/cephtest 2024-03-26T22:00:37.568 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T22:00:37.578 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T22:00:37.598 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T22:00:37.625 DEBUG:teuthology.nuke:shortname: smithi088 2024-03-26T22:00:37.625 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T22:00:37.647 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 20:58:07.908477', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjfHS/LLm9chp4tTBTNYEzt9Wmx3MTOAbaSjAZ3EqNEQpOeGC9i3M1VGwA/WnEhh0nsYuFbrSoDLvUGYozlMbA='} 2024-03-26T22:00:37.650 DEBUG:teuthology.nuke:shortname: smithi138 2024-03-26T22:00:37.650 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T22:00:37.673 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 20:58:07.906794', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9s/bqBbYrYfYK+CQKNC8yzKiot6LieHixZ+p8b9B3iAaOetGRHE5Fp4D39LhtdAYnQfizNBT9ZfLc4L7zbxvM='} 2024-03-26T22:00:37.702 INFO:teuthology.orchestra.console.smithi088:Power off 2024-03-26T22:00:37.703 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T22:00:37.725 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-03-26T22:00:37.725 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:00:37.740 INFO:teuthology.orchestra.console.smithi138:Power off 2024-03-26T22:00:37.740 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T22:00:37.750 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-03-26T22:00:37.764 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off 2024-03-26T22:00:37.764 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:00:37.788 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-26T22:00:41.751 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:00:41.789 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:00:41.861 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-03-26T22:00:41.899 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-26T22:00:45.866 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:00:45.900 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:00:45.976 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-03-26T22:00:46.008 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-03-26T22:00:46.078 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-03-26T22:00:46.109 INFO:teuthology.orchestra.console.smithi138:Power off completed 2024-03-26T22:00:46.214 INFO:teuthology.run:Summary data: description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} duration: 3180.9937999248505 failure_reason: '"2024-03-26T21:30:00.000178+0000 mon.a (mon.0) 2476 : cluster [WRN] Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_gabrioux@teuthology success: false 2024-03-26T22:00:46.214 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T22:00:46.290 INFO:teuthology.run:FAIL